{"id":4722,"date":"2024-04-18T22:49:14","date_gmt":"2024-04-18T22:49:14","guid":{"rendered":"https:\/\/game.intel.com\/?p=4722"},"modified":"2024-05-29T21:16:37","modified_gmt":"2024-05-29T21:16:37","slug":"wield-the-power-of-llms-on-intel-arc-gpus","status":"publish","type":"post","link":"https:\/\/game.intel.com\/gb\/stories\/wield-the-power-of-llms-on-intel-arc-gpus\/","title":{"rendered":"Wield The Power of LLMs On Intel\u00ae Arc\u2122 GPUs"},"content":{"rendered":"
\n

Easily Run a Variety of LLMs locally with Intel\u00ae Arc\u2122 GPUs<\/h3>\n<\/blockquote>\n\n\n\n

Generative AI has changed the landscape of what\u2019s possible in content creation. This technology has the potential to deliver previously unimagined images, videos and writing. Large language models (LLMs) have been making headlines in the age of AI, allowing anyone to prompt their way to generate song lyrics, get answers to complex physics questions, or draft an outline for a slide presentation. And these AI features no longer need to be connected to the cloud or subscription services. They can run locally on your own PC, where you have full control over the model to customize its output.<\/p>\n\n\n\n

In this article, we\u2019ll show you how to set up and experiment with popular large language models (LLMs) on a PC with the Intel\u00ae Arc\u2122 A770 16GB graphics card. While this tutorial will make use of the Mistral-7B-Instruct LLM, these same steps can be used with a PyTorch LLM of your choosing such as Phi2, Llama2, etc. And yes, with the latest Llama3 model too!<\/p>\n\n\n\n

IPEX-LLM<\/h2>\n\n\n\n

The reason we can run a variety of models using the same base installation is thanks to IPEX-LLM<\/a>, an LLM library for PyTorch. It is built on top of Intel\u00ae Extension for PyTorch<\/a> and contains state-of-art LLM optimizations and low-bit (INT4\/FP4\/INT8\/FP8) weights compression \u2013 with all the latest performance optimizations for Intel hardware. IPEX-LLM takes advantage of the Xe<\/sup>-cores XMX AI acceleration on Intel discrete GPUs like Arc A-series graphics cards for improved performance. It supports Intel Arc A-series graphics on Windows Subsystem for Linux version 2, native Windows environments, and native Linux.<\/p>\n\n\n\n

And because all of this is native PyTorch, you can easily swap PyTorch models and input data to run on an Intel Arc GPU with high performance acceleration. This experiment would not have been complete without a performance comparison. Using the instructions below for Intel Arc and commonly available instructions for the competition, we\u2019ve looked at two discrete GPUs positioned in a similar price segment.<\/p>\n\n\n\n

\"\"\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button><\/figure>\n\n\n\n

For example, when running the Mistral 7B model with the IPEX-LLM library, the Arc A770 16GB graphics card can process 70 tokens per second (TPS), or 70% more TPS than the GeForce RTX 4060 8GB using CUDA. What does that mean? A general rule of thumb is that 1 token is equivalent to 0.75 of a word and a good comparison is the average human reading speed of 4 words per second<\/a> or 5.3 TPS. The Arc A770 16GB graphics card can generate words much faster than the average person can read them!<\/p>\n\n\n\n

Our internal testing shows that the Arc A770 16GB graphics card can deliver this capability and competitive or leading performance across a wide range of models compared to the RTX 4060, making Intel Arc graphics a great choice for local LLM execution.<\/p>\n\n\n\n

Now let\u2019s go to set up instructions to get you started with LLMs on your Arc A-series GPU.<\/p>\n\n\n\n

Installation Instructions<\/h2>\n\n\n\n

We can also refer to this page for setting up the environment: Install IPEX-LLM on Windows with Intel GPU \u2014 IPEX-LLM latest documentation<\/a><\/p>\n\n\n\n

1. Disable integrated GPU in device manager.<\/p>\n\n\n\n

2. Download and install Anaconda<\/a>.<\/p>\n\n\n\n

3. After installation is completed, open the Start menu, search for Anaconda Prompt, run it as administrator, and create a virtual environment using the following commands. Enter each command separately:<\/p>\n\n\n\n

conda create -n llm python=3.10.6 \n\nconda activate llm\n\nconda install libuv\n\npip install dpcpp-cpp-rt==2024.0.2 mkl-dpcpp==2024.0.0 onednn==2024.0.0 gradio\n\npip install --pre --upgrade ipex-llm[xpu] --extra-index-url https:\/\/pytorch-extension.intel.com\/release-whl\/stable\/xpu\/us\/\n\npip install transformers==4.38.0<\/code><\/pre>\n\n\n\n
4. Create a text document named demo.py and save it to C:\\Users\\Your_Username\\Documents or to the directory of your choice.<\/p>\n\n\n\n
5. Open demo.py with your favorite editor and copy the following code sample into it:<\/p>\n\n\n\n
from transformers import AutoTokenizer\nfrom ipex_llm.transformers import AutoModelForCausalLM\nimport torch\nimport intel_extension_for_pytorch\n\ndevice = \"xpu\" # the device to load the model onto\n\nmodel_id = \"mistralai\/Mistral-7B-Instruct-v0.2\" # huggingface model id\n\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nmodel = AutoModelForCausalLM.from_pretrained(model_id, load_in_4bit=True, torch_dtype=torch.float16)\nmodel = model.to(device)\n\nmessages = [\n    {\"role\": \"user\", \"content\": \"What is your favourite condiment?\"},\n    {\"role\": \"assistant\", \"content\": \"Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!\"},\n    {\"role\": \"user\", \"content\": \"Do you have mayonnaise recipes?\"}\n]\n\nencodeds = tokenizer.apply_chat_template(messages, return_tensors=\"pt\")\n\nmodel_inputs = encodeds.to(device)\nmodel.to(device)\n\ngenerated_ids = model.generate(model_inputs, max_new_tokens=1000, do_sample=True)\ndecoded = tokenizer.batch_decode(generated_ids)\nprint(decoded[0])<\/code><\/pre>\n\n\n\n
Code built from the sample code in this repository<\/a>.<\/em><\/p>\n\n\n\n
6. Save demo.py. In Anaconda, navigate to the directory that demo.py is located by using the cd command, and run the following command in the Anaconda prompt:<\/p>\n\n\n\n
python demo.py<\/code><\/pre>\n\n\n\n
Now you can get a nice recipe for making mayonnaise!<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Changing Models<\/h2>\n\n\n\n
Using the same environment that we set up above, you can experiment with other popular models on Hugging Face such as llama2-7B-chat-hf, llama3-8B-it, phi-2, gemma-7B-i, and stablelm2 by replacing the Hugging Face model id above in demo.py.<\/p>\n\n\n\n
model_id = \"mistralai\/Mistral-7B-Instruct-v0.2\" # huggingface model id\n\nto\n\nmodel_id = \"stabilityai\/stablelm-2-zephyr-1_6b\" # huggingface model id<\/code><\/pre>\n\n\n\n
Different models might require a different version of transformers package, if you run into errors when launching demo.py, follow the below steps to upgrade\/downgrade transformers:<\/p>\n\n\n\n
    \n
  1. Open Anaconda Prompt<\/li>\n\n\n\n
  2. conda activate llm<\/li>\n\n\n\n
  3. pip install transformers==4.37.0<\/li>\n<\/ol>\n\n\n\n
    Verified transformers versions:<\/strong><\/p>\n\n\n\n
    Model ID<\/th>Transformers package versions<\/th><\/tr><\/thead>
    meta-llama\/Llama-2-7b-chat-hf<\/td>4.37.0<\/td><\/tr>
    meta-llama\/Meta-Llama-3-8B-Instruct<\/td>4.37.0<\/td><\/tr>
    stabilityai\/stablelm-2-zephyr-1_6b<\/td>4.38.0<\/td><\/tr>
    mistralai\/Mistral-7B-Instruct-v0.2<\/td>4.38.0<\/td><\/tr>
    microsoft\/phi-2<\/td>4.38.0<\/td><\/tr>
    google\/gemma-7b-it<\/td>4.38.1<\/td><\/tr>
    THUDM\/chatglm3-6b<\/td>4.38.0<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
    Memory requirements may vary per model and framework. For the Intel Arc A750 8GB running with IPEX-LLM, we recommend using Llama-2-7B-chat-hf, Mistral-7B-Instruct-v0.2, phi-2 or chatglm3-6B.<\/p>\n\n\n\n
    Implementing a ChatBot WebUI<\/h2>\n\n\n\n
    Now let\u2019s move onto implementing a Gradio chatbot webui for a better experience using your web browser. For more information on implementing an interactive chatbot with LLMs, visit https:\/\/www.gradio.app\/guides\/creating-a-chatbot-fast<\/a><\/p>\n\n\n\n
    1. Create a document named chatbot_gradio.py in your text editor of choice.<\/p>\n\n\n\n
    2. Copy and paste the following code snippet into chatbot_gradio.py:<\/p>\n\n\n\n
    import gradio as gr\nimport torch\nimport intel_extension_for_pytorch\nfrom ipex_llm.transformers import AutoModelForCausalLM\nfrom transformers import AutoTokenizer, StoppingCriteria, StoppingCriteriaList, TextIteratorStreamer\nfrom threading import Thread\n\nmodel_id = \"mistralai\/Mistral-7B-Instruct-v0.2\" \n\ntokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)\nmodel = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, optimize_model=True, load_in_4bit=True, torch_dtype=torch.float16)\nmodel = model.half()\nmodel = model.to(\"xpu\")\nclass StopOnTokens(StoppingCriteria):\n    def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool:\n        stop_ids = [29, 0]\n        for stop_id in stop_ids:\n            if input_ids[0][-1] == stop_id:\n                return True\n        return False\n\ndef predict(message, history):\n    stop = StopOnTokens()\n    history_format = []\n    for human, assistant in history:\n        history_format.append({\"role\": \"user\", \"content\": human })\n        history_format.append({\"role\": \"assistant\", \"content\":assistant})\n    history_format.append({\"role\": \"user\", \"content\": message})\n\n    prompt = tokenizer.apply_chat_template(history_format, tokenize=False, add_generation_prompt=True)\n    model_inputs = tokenizer(prompt, return_tensors=\"pt\").to(\"xpu\")\n    streamer = TextIteratorStreamer(tokenizer,  skip_prompt=True, skip_special_tokens=True)\n    generate_kwargs = dict(\n        model_inputs,\n        streamer=streamer,\n        max_new_tokens=300,\n        do_sample=True,\n        top_p=0.95,\n        top_k=20,\n        temperature=0.8,\n        num_beams=1,\n        pad_token_id=tokenizer.eos_token_id,\n        stopping_criteria=StoppingCriteriaList([stop])\n        )\n    t = Thread(target=model.generate, kwargs=generate_kwargs)\n    t.start()\n\n    partial_message = \"\"\n    for new_token in streamer:\n        if new_token != '<':\n            partial_message += new_token\n            yield partial_message\n\ngr.ChatInterface(predict).launch()<\/code><\/pre>\n\n\n\n
    3. Open a new anaconda prompt and enter the following commands:<\/p>\n\n\n\n