About running QLora on Gaudiv2 (Related to bitsandbytes packages)

[ryanpark]

Hi, I have a question while running qlora on Intel Gaudi V2.

Currently, I'm trying to run qlora on HPU architectures (related to https://github.com/artidoro/qlora)

However, I have an issue with the error messages below.

It seems to be related to an error about bitsandbytes packages, and it seems to be bits and bytes only implemented for GPU and XPU (not for HPU).
Is it impossible to run qlora on HPU for now?

Here are my error messages. 

File "/root/qloar_hpu/qlora/qlora.py", line 712, in train
model, tokenizer = get_accelerate_model(args, checkpoint_dir)
File "/root/qloar_hpu/qlora/qlora.py", line 319, in get_accelerate_model
model = AutoModelForCausalLM.from_pretrained(
File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/auto_factory.py", line 493, in from_pretrained
return model_class.from_pretrained(
File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 2903, in from_pretrained
) = cls._load_pretrained_model(
File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 3260, in _load_pretrained_model
new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model(
File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 725, in _load_state_dict_into_meta_model
set_module_quantized_tensor_to_device(
File "/usr/local/lib/python3.10/dist-packages/transformers/utils/bitsandbytes.py", line 99, in set_module_quantized_tensor_to_device
new_value = bnb.nn.Params4bit(new_value, requires_grad=False, **kwargs).to(device)
File "/usr/local/lib/python3.10/dist-packages/bitsandbytes/nn/modules.py", line 331, in to
return self._quantize(device)
File "/usr/local/lib/python3.10/dist-packages/bitsandbytes/nn/modules.py", line 296, in _quantize
w_4bit, quant_state = bnb.functional.quantize_4bit(
File "/usr/local/lib/python3.10/dist-packages/bitsandbytes/functional.py", line 1231, in quantize_4bit
lib.cquantize_blockwise_bf16_nf4(*args)
AttributeError: 'NoneType' object has no attribute 'cquantize_blockwise_bf16_nf4'

Currently, I'm running vault.habana.ai/gaudi-docker/1.17.1/ubuntu22.04/habanalabs/pytorch-installer-2.3.1:latest docker and using 

bitsandbytes==0.45.0

transformers==4.31.0

peft==0.4.0

accelerate==0.21.0

einops==0.6.1

evaluate==0.4.0

scikit-learn==1.2.2

sentencepiece==0.1.99

Thanks.

[James_Edwards · Accepted solution]

The bitsandbytes module currently only supports CUDA based accelerators. That is documented at this link: https://huggingface.co/docs/bitsandbytes/en/installation

.

Gaudi support must be added by that module.