ChatLLM is a VSCode extension for interacting with LLM APIs in a flexible and long-form manner. It leverages the VSCode notebook support to do so, creating a new type of notebook (.chatllm) files where you can interact with an (API-based) LLM system over a long document. Note: this plugin requires you to have your own OpenAI API key (or other LLM API key); it does not work with the free version of ChatGPT.
For the most part, I think that people are using LLMs suboptimally, especially as far as integration into IDEs goes. There are no shortage of "talk to ChatGPT"-type plugins for VSCode, and GitHub Co-pilot has chat functionality too, but these methods all seem to prioritize short-term one-off questions and code generation rather than long form interaction.
In my experience, the best way to leverage LLMs (for most tasks, but very much thinking of coding as the primary example) is via long-form interaction, thinking of chat logs as "projects" that evolve over long periods of time. But at the same time, most public interfaces for LLMs are set up quite badly for this: many can't handle long projects at all (they fail to handle truncation, etc), and even the ChatGPT interface, which I usually wind up using, gets extremely laggy as you build a long chat.
Fortunately, the IDE we're virtually all using already has a perfectly good interface to accomplish this goal: the notebook interface. This plugin defines a new notebook type, the ".chatllm" notebook, that lets you interact with an LLM over long-form discussion using this interface. I'm writing this as a VSCode plugin instead of e.g. a Jupyter notebook cell magic, (which does also exist incidentally), largely because I want to do development inside an actual IDE. And while it might make sense to embed these into Jupyter notebooks with an actual Python kernel, in practice I don't want most development work to happen in a notebook, and would rather have a more specific interface that only handles LLMs.
Installation and setup
Install the extension from the VSCode marketplace. Alternative, you can clone this repo and install the
(replacing X.X.X with the current version of the file).
Setting up models
Once you installed the extension, you need to set up the models to use. To do so, run the
One particularly important settings is the "Truncate tokens" setting. This will truncate chat history to be at most this many tokens prior to the current message (though always still including system prompts and the last user's input, with the ability to not count the system prompt at all toward the total token count). This is necessary for models that have a short history, but also important for models that have a longer context, so that you don't wind up spending too much on queries. Note that this is not a precise truncation: for simplicity, I use the tiktoken tokenizer to count tokens regardless of which model is selected, so the exact token count will differ from the truncation. And the precise mechanism is that we truncate cells after the first one that exceeds the truncation count. So if your model has a low context size, you should leave some buffer.
Using cells and models
Now that you've installed and set up models, you can create or open a `.chatllm`` file. You can add a new code cell and click the Language in the bottom right corner to select the cell type. There are three types of supported cells:
To select a model, click on a "ChatLLM" cell and click on the three dots, and choose the "ChatLLM: Select Model" item. This will bring up a selection list to choose the model you want to use for that cell (from the list of models you created in the Settings view).
The currently-selected model will display in the status bar at the bottom of the VSCode window (I wish it was possible to have this in the cell, where
Once this is set up, to send the chat command just run the cell. It will display the output in markdown format in the cell.
The mechanism of ChatLLM is that all content before the cell you are running will be sent to the API as a chat interaction (truncated at the appropriate length of tokens you specify). As mentioned, Markdown cells are treated as user inputs, with one exception. If you want to edit the previous output of an LLM for future interactions (which is possible, of course), you can select the cell and choose the "ChatLLM: Detach Output" command from the same ellipses menu as you used to select the model. This will detach the cell output into a separate Markdown cell with the starting identifier
Dynamic file inclusion in prompts
To include the contents of a file in your prompt, put the text
ChatLLM is written as a basic VSCode extension, that includes:
The following issues are known: