Hugging Face Course (4/6) – Local Image Generation

/ Computer Science, LLM / By

👉 Back to the Full Course on local models and Hugging Face (+Videos)

Welcome back to part 4! This time we will be looking at other non-LLM models that we can run locally on our own machine. We’ll be stepping away from Ollama and switching to the HuggingFace machine-learning web community from now on.

I’ve deliberately chosen a variety of different high-quality models that require different installation methods and have different quirks and pitfalls to them. My hope is that after seeing the various ways to install and run these models you will be able to tackle installing any other model you want to run in the future on your own.

Hugging Face is a leading platform and community for open-source natural language processing (NLP) and machine learning models. It has a vast library of pre-trained models for NLP tasks like text generation, translation, summarization, question answering, etc.

Researchers and developers can share, discover, and deploy machine learning models and the website provides model cards with details on performance, training data, intended use cases, and licenses.

Hugging Face has a vibrant open-source community contributing code, Hugging Face Spaces (model hosting), Datasets, Tokenizers, and more. As it has a large model library and active community, Hugging Face has become a central hub for the open-source AI Machine Learning community.

So without further ado, let’s head over to the Hugging Face website and see what we can do with it:

Before we head over to the Models section, which is where our focus will be for the rest of these tutorials, I just want to point out that Hugging Face also has an excellent section for Datasets:

You can find all sorts of useful datasets in many categories to use for training, fine-tuning, or testing models. Finally, if we move over one more tab to the right we have Hugging Face Spaces, where you can see all sorts of cool live demos for AI models.

Finding a model

At any point, you may need to sign up for a free account to access some of the features on the website. I recommend just creating a free account before continuing with this tutorial. When you’re done, click on the Models tab, and let’s get started!

We have so many models available, from Computer Vision to LLMs to audio-related models! First click on the Text-to-Image option on the left side. Sort the results by Most downloads and you should see a model named stabilityai/sdxl-turbo somewhere near the top.

SDXL Turbo is made by the stability.ai team and is based on Stable Diffusion XL. It is a pretty fast model that we can deploy locally to generate images at will! Click on the model to open the model card with more information:

Here you can see all the details for this model. It has a whopping 2.3 million downloads in the last month alone! If you scroll down you will see instructions on how to install this model and usually some example code:

These are the base instructions I will be using to figure out how we can run the model locally, but in some cases, we’ll also have to search the GitHub repository for a certain model to find more detailed instructions.

Finally, if we scroll down even further we can see some of the limitations of this model, and also that we’re allowed to use this for commercial purposes!

Jupyter Notebooks

For this part of the tutorial, we’ll be using Jupyter Notebooks, as it’s more practical to code interactively and keep our Python kernel running while we add more code. It will also display our generated images very nicely which is the main benefit here, as we cannot display image output in the console.

Install the Jupyter Notebooks extension for VS Code if you don’t have it already, by going to the extensions tab and searching for Jupyter Notebooks. It’s the extension with an insanely high number of downloads. (Yupyter by Microsoft).

I won’t be going into a detailed explanation of Jupyter Notebooks here, but if you’re not quite familiar with them, don’t worry. Just follow along with the video version of this tutorial so you can follow along step-by-step and you’ll get a feel for how it works.

First, go ahead and create a new file named image_gen.ipynb in the root folder of your project:

📁Local_Models
    📁test_files
    📄chat_app.py
    📄image_gen.ipynb    ✨New file
    📄local_chat.py
    📄local_chat_memory.py
    📄memory.py
    📄model_preloader.py
    📄Pipfile
    📄Pipfile.lock

Installing PyTorch

Now we’ll need to install PyTorch. PyTorch is an open-source machine learning library based on the Torch library, used for building and training neural networks. PyTorch supports hardware acceleration like CUDA GPUs and has a large community around it. For many models, you will need to have PyTorch installed to be able to run them.

Here is where it gets a bit tricky though, as some of you will have a CUDA GPU and some won’t. Head over to the PyTorch website and make sure you’re on the Start Locally tab:

Now scroll down until you get to the following choice menu section:

Choose the Stable build version, then pick the platform you’re on. I’ll be going for Windows and will use the Pip package manager inside VSCode as we have done so far. Obviously our language will be Python. Next up is your choice for either CUDA 11.8, CUDA 12.1, or CPU.

If you don’t have a CUDA GPU, CPU is fine. Everything we will do will still run fine, it will just be slower, so go ahead and choose CPU. You will get the command to run in your terminal. Make sure you change it to install inside of our virtual environment by replacing pip3 with pipenv:

#pip3 install torch torchvision torchaudio
pipenv install torch torchvision torchaudio

After running this command, skip ahead a little bit to where we get started with the Jupyter Notebook.

If you do have a CUDA GPU, we have 2 options left. Personally I would just go for the CUDA 12.1 version. There are a couple of potentially confusing points here so let’s go over them:

  • It may seem like you need to install CUDA yourself, but the CUDA version of the command below will actually install the correct runtime dependencies for you. This is different than actually installing the full CUDA toolkit. As we don’t need the full CUDA toolkit here, we won’t bother with all of that, but if you want more information you can find it here.
  • If you have a graphics card but you’re not quite sure whether it’s CUDA compatible, you can check here. If it’s on the list you should be good to go.

Ok, so now we have the install command pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121. If we just change this to pipenv instead, it’s going to get confused by the --index-url part of the install command. There is not really any clear documentation on how to use this particular combination with a pipenv, so we’re just going to cheat a little bit here. (There is a solution here if you’re really interested but I don’t want to bother you too much with the details of pipenv Pipfiles as it’s quite off-topic for this tutorial.)

Even though we are inside the virtual environment, we can still use the pip3 .... install command. You may have to change this to pip instead of pip3 though depending on your system. What will happen if we do is that pip will still install these dependencies inside of our virtual environment, so it’s not a big deal and will work fine, but it’s not the most elegant solution as it will be missing from our Pipfile and Pipfile.lock files. (You can add it to the pipfile as a comment: # run command: pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121)

So go ahead and run the command in your terminal with the virtual environment still activated: