Welcome back to part 5 of this tutorial series. In this part, we’ll take a look at how to run LLMs locally on your own computer for free and also at the HuggingFace community which has loads of models available for you to use in all sorts of fields. As always, we’ll have resources available to go way more in-depth on these topics.
Running LLM models locally
There are many free LLM models out there that you can run locally on your computer. Some of the more well-known models are the LLAMA models. We can very easily run these models on our own machine using a tool called ollama. Despite what the name suggests, this tool can also run many other models besides LLAMA. As usual, let’s just jump right in and we’ll explain as we go along.
Installing ollama
Go to the ollama website and download the appropriate version for your operating system.
I am on Windows, so I’ll be using the Windows (Preview) version here. Once you have downloaded the file, click the installer to get started.
Just let it run and it will do its own thing…
When it’s done, you will see an alert box in the corner:
Running an LLM
We now have the Ollama server running in the background. Let’s tell it what LLM we want it to run. We’ll get started with Llama 3.
Llama 3 is Meta (Facebook)’s open-source large language model. It is a general-purpose chat model just like ChatGPT, meaning it is good at general NLP tasks like question answering, sentiment analysis, text classification, and even code generation. It’s also free and open source!
There are two basic sizes of Llama 3, the smaller one having 8 billion parameters which is quite practical to run on most computers, and the larger one having 70 billion parameters.
To get started all you need is a terminal window. I’ll use my bash terminal integrated into VSCode here as I still have it open from the previous tutorial and we’ll get back to coding soon anyway. Run the following command in a terminal window:
ollama run llama3
When you run this command, Ollama will automatically start pulling the llama 3 model for you. This will take a while as the model is around 4.7 GB in size.
Note that we are downloading the smaller version of Llama 3 here. If you want to run the larger version, replace the above command with ollama run llama3:70b instead, but make sure you have a powerful enough computer as the model alone is 40 GB in size.
As all steps are exactly the same I’ll stick with the smaller version for demonstration purposes.
pulling manifest pulling 00e1317cbf74... 100% ▕████████████████████████████████▏ 4.7 GB pulling 4fa551d4f938... 100% ▕████████████████████████████████▏ 12 KB pulling 8ab4849b038c... 100% ▕████████████████████████████████▏ 254 B pulling 577073ffcc6c... 100% ▕████████████████████████████████▏ 110 B pulling ad1518640c43... 100% ▕████████████████████████████████▏ 483 B verifying sha256 digest writing manifest removing any unused layers success >>> Send a message (/? for help)
So let’s go and send it a message, shall we?
>>> Where does the sad frog meme come from?
Give it a second and it should start streaming your response to the terminal. This may be pretty fast or very slow depending on how powerful your system is. Ollama will use your GPU if available or run in CPU-only mode if required.
And there’s our answer:
The Sad Frog, also known as "Pepe" (yes, it's a guy!), is a beloved internet meme that originated on 4chan's /b/ board in the early 2010s. Here's a brief history: * In 2008, artist Matt Furie created Pepe, a cartoon character for his comic book series "Boys LoveGirls" (yes, it's a play on words!). Initially, Pepe was just a normal frog with a big smile and googly eyes. * Around 2012-2013, /b/ board users began to manipulate and modify Pepe's facial expressions to create the sad version. This was done by altering his eyebrows, mouth, or adding tears, making him appear dejected and melancholic. * The Sad Frog quickly gained popularity as a meme, symbolizing disappointment, sadness, or just general unhappiness. ... cut for brevity ...
It was longer and very informative, but the purpose here is not to learn about our sad green froggy friend, so I cut it short! For a reasonably small model running on our own computer, this is pretty impressive! This stuff is not coming from the internet, it’s coming from the model running on our own computer.
When you’re done chatting with your new local LLM friend, you can type the following in the terminal to end the session:
/bye
There are many other models we can run with Ollama, all with differing sizes and capabilities. It’s also possible to communicate with the Ollama server running on your machine using rest API. This means we can programmatically interact with it and also use our LangChain knowledge to chain together and build cool automated stuff on top of this.
There will be a link at the end of this lesson with a course that has all of that and more in it for you if you’re interested in learning more.
HuggingFace
We’ll be stepping away from Ollama and switching to the HuggingFace machine-learning web community for now. This opens up a whole world of other non-LLM AI models!
Hugging Face is a leading platform and community for open-source natural language processing (NLP) and machine learning models. It has a vast library of pre-trained models for tasks like text generation, translation, summarization, question answering, etc. However, it also has many other types of models like image generation, audio generation, and even models for tasks like object detection.
Researchers and developers can share, discover, and deploy machine learning models and the website provides model cards with details on performance, training data, intended use cases, and licenses.
Hugging Face has a vibrant open-source community contributing code, Hugging Face Spaces, Datasets, and more. As it has a large model library and active community, Hugging Face has become a central hub for the open-source AI Machine Learning community.
So without further ado, let’s head over to the Hugging Face website and see what we can do with it: