Local LLM on Intel Arc A770 16 Gb

[Roflosaur]

Strange behavior I see, when use GPU as computing device for language neural networks. They "goes insane", and they responses to sentence "introduce yourself" are completely different than on CPU. I tested "fast models", as GPT4All Falcon and Mistral OpenOrca, because for launching "precise", like Wizard 1.2 is impossible because too low video memory. So, on CPU all works fine, but on GPU LLM's goes crazy. Screenshots in attach.

CPU: 12th Gen Intel(R) Core(TM) i5-12400F 2.50 GHz

RAM: 32,0 Gb

GPU: Intel Arc A770 16 Gb

OS: Windows 10 2H22 build 19045.2965

Driver version: 31.0.101.4952

Motherboard: MSI PRO-B660M-P-DDR4

BIOS: 7D24v2D

ReBar: on

[Jean_Intel]

Hello Roflosaur,

Thank you for posting in the Intel Communities. We see that you are experiencing issues with language neural networks when you use the Intel Arc A770 as the computing Device. We would be more than glad to help you.

• How are you changing the computing device between the CPU and GPU?
• We noticed that there is an ongoing conversation when you ask the program to "introduce yourself." Do you receive the same response if you start a new conversation with the GPU as the computing device?
• Are you developing or working on a project related to the language neural networks? We would like to know if you are part of a developing team, so we can better assist you on this matter.
• We understand that you have an Intel Arc A770: however, we would like to know the exact graphics card model. Are you using a Limited Edition Card, or are you using any card from a different manufacturer: Acer, AsRock, Sparkle, MSI?
• We would like to have more information about your system. We would like to request you to share a system report using the Intel System Support Utility (Intel SSU):
  • Open the application and click "Scan" to see the system and device information. By default, Intel SSU will take you to the "Summary View."
  • Click on the menu where it says: "Summary" to change it to "Detailed View."
  • To save your scan: click "Next"; then "Save."

Best regards

Jean O.

Intel Customer Support Technician

[Roflosaur]

1. See 1.png

2. No, CPU fine and GPU works incorrect. I suppose because it's uses CUDA by default (and probably built only for), but I'm not sure. More information about GPT4All you can find here:  https://gpt4all.io/index.html and https://home.nomic.ai/

3.I'm not developer, just a user that copy and paste commands from GitHub by guides, if needed.

4.Limited Edition.

5.See scan2.txt

[Jean_Intel]

Hello Roflosaur,

Thank you for the information provided, based on the inforamtion you have provided, we will proceed to look into this matter internally. Let us look into this, and we will be posting back as soon as we have more details.

Best regards

Jean O.

Intel Customer Support Technician

[Jean_Intel]

Hello Roflosaur,

Thank you for your patience, waiting for a response on this matter. After looking into this matter, we recommend you to contact the app developer since CUDA is not supported by Intel Arc graphics products, plus there is no sign that the LLM tools are optimized for the Intel Arc GPUs.

Best regards

Jean O.

Intel Customer Support Technician

[Jean_Intel]

Hello Roflosaur,

As we have not heard from you, we will proceed to close this thread. Remember that we recommend you contact the app developer since CUDA is not supported by Intel Arc graphics products. If you need any additional information, submit a new question, as this thread will no longer be monitored.

Best regards

Jean O.

Intel Customer Support Technician