Creating a GitHub Copilot Extension Agent to Chat with Your Data

The idea of leveraging GenAI to become more productive has become widespread in recent years. The base knowledge and training data of GenAI models are already so useful that many knowledge workers, such as developers, have experienced productivity improvements even when the model is not trained on their specific data. However, this doesn’t stop organizations from trying to further improve productivity by looking for ways to integrate their own data with these GenAI applications.

GitHub Copilot is no exception. In addition to allowing developers to choose the model that works best for them, GitHub has introduced features that integrate GitHub Copilot with your own data, such as indexing repositories for Copilot Chat and fine-tuned models for Copilot Enterprise.

This post is about one of these new features: GitHub Copilot Extensions. In this simple example, I deployed an Azure OpenAI Chat with Your Data Solution Accelerator. This allows me to upload documents (e.g., .docx, .pdf) that are vector-indexed and stored in a source outside of GitHub. I then created a simple GitHub Copilot Extension that calls the APIs from this solution accelerator, enabling me to chat with my documents directly from the GitHub Copilot Chat.

GitHub Copilot Extensions vs. VS Code Chat Extensions

Before we start, it’s important to note that there are two ways to extend GitHub Copilot Chat. The first way is to create a VS Code Chat Extension, which essentially adds a @chat-participant to the GitHub Copilot Chat panel in VS Code. This method is great if you want your extension to perform VS Code-specific features and can be published in the VS Code Marketplace. One of the biggest benefits of this approach is that it does not require a GitHub organization to work.

The second way is to create GitHub Copilot Extensions, which are a type of GitHub App and can be published to the GitHub Marketplace. The main advantage is that GitHub Copilot Extensions work on a variety of supported clients and IDEs, unlike VS Code Chat Extensions which only work in VS Code. However, because they are a type of GitHub App, you will need a GitHub organization to install your extension (GitHub free orgs, GitHub teams, and GitHub Enterprise Cloud orgs are all supported, but GitHub Copilot Standalone accounts are not). This is the approach we will take in this post.

The Big and Simple Picture: Creating a Copilot Extension

As of this writing, the documentation for building a GitHub Copilot extension is not yet comprehensive. I found the best way to learn is to review the code samples available in this GitHub repository. Here, I discovered that the implementation is extremely simple:

sequenceDiagram
    IDE->>+GitHub App: What are our coding standards?
    Note over GitHub App,MyExtension-API: Request Payload
    GitHub App->>MyExtension-API: 
    MyExtension-API->>ChatWithYourData-API: What are our coding standards?
    ChatWithYourData-API-->>MyExtension-API: Our coding standards are...
    Note over MyExtension-API,GitHub App: Response Payload + DONE Event
    MyExtension-API-->>GitHub App: 
    GitHub App-->>-IDE: Our coding standards are...

Simply put, a GitHub Copilot Extension is a custom API that receives chat messages in a specific JSON format and then responds with another JSON format. What you do with the message is up to you—it could be as simple as echoing the message or as complex as calling your own GenAI model with custom prompts. The diagram below shows how a GitHub Copilot Extension is configured. The API URL in the diagram represents my extension API that is deployed in Azure Functions.

Because it is custom code that handles and processes the chat data, it is the extension creator’s responsibility to ensure the data is handled securely and complies with all relevant regulations.

My GitHub Copilot Extension: Chat with Your Data

This is the code for my GitHub Copilot extension. Feel free to check it out directly, as I’ve included instructions on how to get started in the README file. If you’d like to read more details, continue on.

Prerequisite: Deploy the Chat with Your Data Solution Accelerator

I wanted an extension that allows me to chat with my .docx and .pdf files. The simplest way to achieve this is by deploying the Chat with your Data Solution Accelerator. While the web chat interface isn’t necessary, the admin interface simplifies the process of uploading documents that are automatically indexed for RAG-pattern queries.

After deploying this to Azure, you can find the conversation API endpoint at https://web-<uniqueid>.azurewebsites.net/api/conversation.

Logic 1: Passing Chat Messages to the Solution Accelerator

Through debugging the sample Copilot extension code, we find that the request chat message payload are in the standard ChatGPT messages format:

{
    "copilot_thread_id": "326fdf6e-a3f9-4021-8848-8ec76a3afd9f",
    "messages": [
        {
            "role": "user",
            "content": "hi"
        }
    ]
}

All that’s needed is to convert this to the payload that https://web-<uniqueid>.azurewebsites.net/api/conversation expects:

{
    "conversation_id": "326fdf6e-a3f9-4021-8848-8ec76a3afd9f",
    "messages": [
        {
            "role": "user",
            "content": "hi"
        }
    ]
}

Logic 2: Parse Response

The response payload from the solution accelerator is in the following format, and all we need to do is to extract the content field from the last messsage:

{
  "choices": [
    {
      "messages": [
        {
          "content": "{\"citations\": [], \"intent\": \"hi\"}",
          "end_turn": false,
          "role": "tool"
        },
        {
          "content": "Hello! How can I assist you today?",
          "end_turn": true,
          "role": "assistant"
        }
      ]
    }
  ],
  "created": "response.created",
  "id": "response.id",
  "model": "gpt-4o",
  "object": "response.object"
}

Logic 3: Send Response to GitHub Copilot Chat

This is where it gets interesting. There are five possible response events that GitHub Copilot Extensions support:

1. a text response,
2. an acknowledgement response,
3. a confirmation response,
4. a reference response, and
5. an error response

These response events can be seen in the current Javascript Preview SDK. Note: Since I am using Python for my code, I used GitHub Copilot to translate the Javascript code to Python, see here.

For this simple experiment, I used the text response type. Our extension ends with returning the following response

ghcp_response = {
    "body": create_text_event(last_message) + create_done_event()
}

return ghcp_response_json

Next Steps

The rest is simple, do check out my respository for the full code and the step-by-step instructions for running it.

I hope this simple post inspires you to create your own GitHub Copilot extensions. Since it’s a new feature, there aren’t many great samples available yet. I encourage you to try it out and share your results with the community.