AutoGen Studio (3/4) – Next Level Prompt Engineering

/ OpenAI / By

👉 Go Back to the Full Course: Next Level Prompt Engineering with AutoGen Studio

Hi and welcome back to part 2, where we’re going to be diving head first into AutoGen Studio, now that we have it running inside a secure environment where it can freely execute any code it generates without restrictions or danger.

You will be dropped into the Playground tab which lists your sessions. A session is just sort of an ongoing conversation between you and one of the workflows you’ve created. A quick pro-tip, you can click that sun icon next to the ‘Guest User’ name on the top right to switch to ‘night mode’.

That’s basically it for the Playground tab, so let’s take a quick look at the Gallery:

This is of course empty right now, but when you have a session you want to save from the Playground tab, you can save it to this Gallery tab, and it will remain here for your future reference, even if you delete the session in the Playground tab.

The build tab

Now let’s take a rough look at the Build tab before we dive in deeper.

We have 4 menu items here. The top one is for Skills. These skills are basically just Python functions. If you’ve worked with OpenAI function calling before, these are just the functions or tools that you used to pass in.

The major difference is that traditionally ChatGPT gave us the arguments and the function it wanted to call and we then had to call the functions ourselves. In AutoGen, the AI agents can call these functions or skills themselves, executing code autonomously. They can even generate functions that don’t exist in this skill list and execute them right away. This is why we went through all the trouble of setting up a separate Docker environment to run in!

If you click on one of these skills, you’ll see that it is just a Python function:

Moving on to the Models tab, here we can define the LLMs that AutoGen Studio has access to. We can use these to define our connection to ChatGPT if we want to use different versions, like GPT-3.5 Turbo or GPT-4, but we can also use other LLMs besides OpenAI’s ChatGPT and even run an LLM locally on our own computer and add the model to the list in here. We’ll get back to this topic later.

If you click on one of the models that is predefined, you’ll see that the details are mostly empty:

This doesn’t really matter for now as AutoGen Studio will sort of default to ChatGPT-4 automatically and read our API key that we set as an environment variable using our Dockerfile. So we don’t really have to set up anything in the Models tab for now.

Moving on to the Agents tab, here is where you define the AI agents or ‘virtual members of your team’ or ‘robot slaves’🤖🤖 if you will 🙃.

The two agents that come predefined are the primary_assistant and the userproxy. The primary assistant is a standard assistant that can write code, has a Prompt with instructions it follows, can use Skills, and can be linked to a certain LLM Model or Models.

The userproxy is a special agent in your group that is used to represent the user. It takes your input and passes it to the agent or groups of agents, and is also used to execute code.

We’ll go into more detail on specific prompts and settings for the agents later.

Workflows

The final tab is the ‘Workflows tab’:

This is where we combine it all together to create a Workflow. It can be as simple as just two agents, which is the userproxy agent taking the user input and feeding it to an assistant agent, like this:

    🤖Userproxy --> 🤖Assistant

But it can also be more complex, involving multiple agents, which looks something like this:

🤖Userproxy --> 🤖Group chat manager --> 🤖🤖🤖🤖 (team)

Where the userproxy agent feeds into a group chat manager agent, which then feeds into a whole group of agents that all collaborate to achieve your goal.

So let’s check this stuff out, starting with something simple. Click the Visualization Agent Workflow on the right-hand side:

As you can see here, the overall workflow has several properties we can set:

  • The Workflow Name and Workflow Description are just a short name and description, mostly for our own convenience and to keep things organized.
  • The Summary Method indicates the final return type. As the agents work on our assignment, eventually they will be done. While we will be able to see the whole conversation no matter what setting we use, this Summary Method setting will determine what the agents will return as a final answer to the conversation. We can choose between:
  • last – The last message of the conversation is shown as the outcome of the task.
  • none – No summary is shown, but you can of course check out the whole conversation.
  • llm – This will generate a summary of the entire conversation using the linked LLM model. So it’s like having one final LLM call to ask for a summary.
  • The Sender, which is usually the userproxy agent which receives your input question and sends it to the other agent(s).
  • The Receiver, which is the agent or team that will receive the assignment and work on it. This can be either a single agent or a group of agents.

So go ahead and click on the Sender which is the userproxy agent:

Here in the userproxy agent properties we again have a name and description which are just for our own convenience. The Max Consecutive Auto Reply sets a limit to the number of messages so you can set a limit to how long you want the ‘conversation’ to go on for and not get stuck in some kind of infinite loop just making endless LLM calls.

The Human Input Mode has three settings in the underlying AutoGen library; ALWAYS, TERMINATE, or NEVER, and it determines when the userproxy will ask the human being for feedback during generation (human-in-the-loop). This feature is not yet available in the user-interface AutoGen Studio though, so we’ll just leave it at NEVER for now, which means the models do their work independently without human feedback and return the final result.

If you’re watching this tutorial in the future, and this feature is now in the Studio version of AutoGen, know that the ALWAYS version will always ask for human feedback whenever it receives a message (as the name ‘always’ suggests). TERMINATE will ask for human feedback when it encounters a termination message (more on those later), so it will check with you if you’re satisfied with the final result before making it into the final answer and ending.

If we scroll down below we have the following settings:

The System Message is like the instructions for the agent. For a userproxy agent we just leave these empty. The Model and Skills sections allow us to give the agent access to specific LLM models and skills that we can define in the other two tabs we looked at. Again, for the userproxy agent just leave everything blank as it’s sort of a special agent.

Whenever defining an agent, be careful with that last dropdown at the bottom “Or replace with an existing agent” as it is a quick way to overwrite all the settings you just made with whatever agent you select.

So go ahead and close up the userproxy without changing anything, and then let’s check out the visualization_assistant that is set as the receiver: