If You Are Looking for the AutoGen's UI with Document Retrieval, Here Is It

A Walkthrough of AutoGen+Panel Project Ep.3 - RAG Implementation

---

After the buzz surrounding my little piece of work on the AutoGen+Panel for visualized AutoGen application, I was taken aback by the curiosity and interest of my community friends. Many of them reached out and queried further development into how the system could handle more complicated conversation scenarios, particularly those that document retrieval interactions within the AutoGen conversation flow. Since both of the fundamentals in AutoGen and Panel support this implementation, I’m excited to roll out the highly anticipated update to the RAG feature. This new addition is applied to extend the experience when AutoGen’s multi-agent conversation needs external knowledge to complete a more complex task.

Before we dive in, let’s have a quick experience with a quick demo showcasing what’s now available:

Background

If you’re unfamiliar with my prior tutorials and the AutoGen+Panel project, let me provide you with a brief overview.

AutoGen by Microsoft is an LLM development framework for creating automated conversations between multiple AI agents to complete complex tasks autonomously. It was praised for customizable LLM-powered agents and seamless integration with human input. The framework stands out for offering versatile conversation patterns and being compatible with other models, tools/functions, and prompting technologies, to enhance agent capabilities. However, when using it, I found its output challenging to recognize in the command line, so I prompted the use of Panel’s new Chat Component to create a more accessible chatbot-style UI. My previous tutorials demonstrate the development of an integrated Panel UI working with AutoGen, which makes multi-agent conversations and their outputs including code and tables more user-friendly and readable.

Again, this is AutoGen’s original UI

This is the basic version of the Panel-powered AutoGen UI I created previously.

The previous version demonstrated a demo of a research chat group, where six agents were in the group conversation to plan, modify, refine, and generate concrete results for complex tasks. Once you provide a query, the agents will start to generate a step-by-step context based on the sequence appointed by AutoGen’s core manager, which will be displayed by the Panel framework. You can read that tutorial and try to build one.

For real-world problems, LLM applications always need to resolve issues from external knowledge instead of its knowledge base which is at a limited scale and outdated. As the development of AutoGen, it supports RAG functionalities powered by both OpenAI Assistants API and native document retrieval tools. In this project, I will show you how to create a visualized AutoGen application by Panel to implement the RAG function.

Code Walkthrough

For an easy demonstration of the code, in this demo, I will use a two-agent conversation scenario instead of a group chat so we can focus on how to use Panel to upload the file and insert the file into OpenAI Assistant “Retrieval” and let the agent respond with the file content. Please note that you don’t have to learn from the last article for the fundamental coding, this tutorial will cover all the source code and their explanation of AutoGen and Panel in an efficient way, so you can read this article alone to have quick implementation.

1. Upgrade the dependencies

Let’s first install the packages that are needed for this project. I recommend installing AutoGen from its Git Repository as it’s still in busy development for new features and bug fixing.

pip install openai panel
pip install git+https://github.com/microsoft/autogen.git

2. Setup the agents

Since we are going to implement an agent with document retrieval functionality, let’s create an LLM config with related parameters:

import autogen
import panel as pn
import openai
import os
import time
import asyncio
from autogen.agentchat.contrib.gpt_assistant_agent import GPTAssistantAgent
from autogen import UserProxyAgent
from openai import OpenAI

os.environ["OPENAI_API_KEY"] = "sk-Your_OpenAI_Key"
assistant_id = os.environ.get("ASSISTANT_ID", None)

client = OpenAI()
config_list = [
    {
        'model': 'gpt-4-1106-preview',
    }
    ]
llm_config = {
    "config_list": config_list,
    "seed": 36,
    "assistant_id": assistant_id,
    "tools": [
            {
                "type": "retrieval"
            }
        ],
    "file_ids": [],

In this configuration, you could provide your OpenAI Assistant ID from the environment variable ASSISTANT_ID pointing to an assistant you’ve already created or let AutoGen create one initially by not providing an ID. Make sure the type of tools should be retrieval. The program will ask the user to upload a file on the Panel UI later so the file_ids can be left empty at this stage.

Now we can initiate the two agents we need, one is the user proxy that allows a human user to input directions after every time the assistant responds. Like what we do in the last tutorial, to enable human input from Panel Chat Input widgets to AutoGen’s core processing, we have to create a specific agent class MyConversableAgent for this user proxy that can re-direct the input from the original command line to the Panel input widget.

input_future = None

class MyConversableAgent(autogen.ConversableAgent):

    async def a_get_human_input(self, prompt: str) -> str:
        global input_future

        chat_interface.send(prompt, user="System", respond=False)
        # Create a new Future object for this input operation if none exists
        if input_future is None or input_future.done():
            input_future = asyncio.Future()

        # Wait for the callback to set a result on the future
        await input_future

        # Once the result is set, extract the value and reset the future for the next input operation
        input_value = input_future.result()
        input_future = None
        return input_value


user_proxy = MyConversableAgent(name="user_proxy",
    code_execution_config=False,
    is_termination_msg=lambda msg: "TERMINATE" in msg["content"],
    human_input_mode="ALWAYS")

Just remember the input_value in the class function a_get_human_input() is the text body that inputs from Panel. The chat_interface.send() function is called to display the prompt message AutoGen generates to ask the user to input something. The Panel widget chat_interface will be initiated in late steps.

Prompt message for human input

Then, don’t forget to create the GPT-based Assistant agent with LLM capability, plus OpenAI’s retrieval assistant. There is no effort required to call OpenAI Assistant APIs as the GPTAssistantAgent class wraps them for you implicitly.

gpt_assistant = GPTAssistantAgent(name="assistant",
    instructions="You are adept at question answering",
    llm_config=llm_config)