Track Large Language Models¶

When it comes to building applications using Large Language Models, a lot of time is spent working on prompt engineering rather than training models. This new workflow requires a different set of tools that Comet is developing under the umbrella of LLMOps.

The tools available for Large Language Models fall under two categories:

• LLM projects: Dedicated views specifically for analyzing prompts, responses and chaings. The functionality has been specifically built to allow you to analyze tens of thousands of prompts.
• LLM panels: Visualizations that can be used with Experiment Management to view prompts and chains, usefull when projects contain both fine-tuning and prompt engineering use-cases.

LLM Projects¶

LLM Projects differ from standard Experiment Management projects in that they are customized to the prompt engineering workflow rather than a fine-tuning or model training workflow.

Creating an LLM project can be done in one of two ways:

• Through the Comet UI by using the New Project button and choosing Large Language Model as the project type.
• By using the LLM Python SDK to log a prompt to a new project.

Once the LLM project has been created, it can be accessed from the Projects page and gives you access to a number of prompt analysis tools.

Learn more about LLM projects here.

LLM Panels¶

When working with LLMs, some workflow are not just focused on prompt engineering but include a combination of fine-tuning and prompt engineering. This workflows can be tracked by using two different projects for each parts of the workflow or by using a single project and utilizing the LLM panels.

There are two LLM panels that are currently available:

• Prompt Playground: Interact with Large Language Models from the Comet UI. All your prompts and responses will be automatically logged to Comet.
• Prompt History: Track all your prompt / response pairs. You can also view prompt chains to identify where issues might be occuring.

Learn more about LLM panels here.

Example projects¶

A demo LLM project is available here: LLM demo project

A demo project with the Prompt History panel is available here: LangChain demo project.