👉 Back to the Full Course on local models and Hugging Face (+Videos)
Hi and welcome to this tutorial series on running Large Language and Machine Learning Models on your local machine. In this first part, we’ll be looking at running LLMs locally on our own computer for free. To achieve this we will use ollama which is a tool that is designed to help us achieve just this. As usual, let’s just jump right in and we’ll explain as we go along.
Installing ollama
Go to the ollama website and download the appropriate version for your operating system.
I am on Windows, so I’ll be using the Windows (Preview) version here. Once you have downloaded the file, click the installer to get started.
Just let it run and it will do its own thing…
When it’s done, you will see an alert box in the corner:
Running an LLM
We now have the Ollama server running in the background. Let’s tell it what LLM we want it to run. We’ll get started with Llama 3.
Llama 3 is Meta (Facebook)’s open-source large language model. It is a general-purpose chat model just like ChatGPT, meaning it is good at general NLP tasks like question answering, sentiment analysis, text classification, and even code generation.
Unlike OpenAI GPT-4 and Google Gemini, Llama 3 is freely available for almost all uses and is open-source.
There are two basic sizes of Llama 3, the smaller one having 8 billion parameters which is quite practical to run on most computers, and the larger one having 70 billion parameters.
To get started all you need is a terminal window. I’ll use my bash terminal integrated into VSCode here as we’ll be using VSCode for our coding in a moment anyway.
ollama run llama3
When you run this command, Ollama will automatically start pulling the llama 3 model for you. This will take a while as the model is around 4.7 GB in size.
Note that we are downloading the smaller version of Llama 3 here. If you want to run the larger version, replace the above command with ollama run llama3:70b instead, but make sure you have a powerful enough computer as the model alone is 40 GB in size.
As all steps are exactly the same I’ll stick with the smaller version for now.
You should see something like the following:
pulling manifest pulling 00e1317cbf74... 100% ▕████████████████████████████████▏ 4.7 GB pulling 4fa551d4f938... 100% ▕████████████████████████████████▏ 12 KB pulling 8ab4849b038c... 100% ▕████████████████████████████████▏ 254 B pulling 577073ffcc6c... 100% ▕████████████████████████████████▏ 110 B pulling ad1518640c43... 100% ▕████████████████████████████████▏ 483 B verifying sha256 digest writing manifest removing any unused layers success >>> Send a message (/? for help)
So let’s go and send it a message, shall we?
>>> What is a main sequence star?
Give it a second and it should start streaming your response to the terminal. This may be pretty fast or very slow depending on how powerful your system is. Ollama will use your GPU if available or run in CPU-only mode if required.
A Main Sequence Star!
In the vast universe, stars come in different forms and stages of life. A Main Sequence Star (MSS) is a specific type of star that's incredibly
important for our existence.
**What is a Main Sequence Star?**
A Main Sequence Star is a star that's fusing hydrogen atoms into helium in its core. This process releases an enormous amount of energy, making the
star shine brightly.
Here are some key characteristics of Main Sequence Stars:
1. **Stable burning**: MSS stars have a stable, consistent energy output, which means their brightness remains relatively constant over millions of
years.
2. **Hydrogen fusion**: The core of the star is where hydrogen atoms (protons) combine to form helium, releasing energy in the form of light and
heat.
3. **Medium mass**: Main Sequence Stars typically have a mass between 0.1 and 10 times that of our Sun (G-type star). More massive stars are red
supergiants or blue giants.
4. **Surface temperature**: MSS temperatures range from about 3,000 to 50,000 Kelvin (4,500°F to 90,000°F), which determines the star's color:
white, yellow, orange, or red.
**Why Main Sequence Stars matter**
Our Sun is a G-type Main Sequence Star, and it's the perfect distance from us for life as we know it. The energy released by our Sun (and other MSS
stars) powers:
1. **Photosynthesis**: Plants convert sunlight into chemical energy through photosynthesis.
2. **Earth's climate**: The stable energy output of our Sun regulates Earth's temperature and weather patterns.
3. **Life on Earth**: The warmth and light provided by the Sun support life as we know it.
In summary, Main Sequence Stars are stars that fuse hydrogen into helium in their cores, releasing energy and supporting life across the universe!
Do you have any more questions about Main Sequence Stars or anything else?
Pretty good, you are now running llama 3 locally and chatting with it inside your terminal! Play around with it as you like and then type the following to exit:
/bye
Running other models
So let’s take a look at running some other models. We’ll start with Phi-3. Run the following command to pull the model to your computer (the run command did this automatically for us before):
ollama pull phi3
Phi-3 is a family of open AI models developed by Microsoft that are small language models (SLMs). Microsoft claims that Phi-3 models are the most capable SLMs available, outperforming models of the same size and the next size up in language, coding, and math capabilities.
Phi-3 models are designed to be more efficient and cost-effective than large language models (LLMs), while still delivering strong capabilities. They are designed to be more accessible and deployable on resource-constrained devices like smartphones. This comes at the cost of less factual knowledge and only dealing with the English language well.
The download will finish reasonably quickly, as the model is only 2.3 GB in size. Once it’s done, run the following command to check our locally saved models:
ollama list
And you will see we now have two models available:
NAME ID SIZE MODIFIED llama3:latest a6990ed6be41 4.7 GB 25 minutes ago phi3:latest a2c89ceaed85 2.3 GB 3 seconds ago
Now let’s run phi3:
ollama run phi3
Then ask it a question to check it out (if you need multiple lines wrap them in """):