Record actual model used to run the prompt about llm HOT 7 CLOSED

Current schema:

% sqlite-utils schema "$(llm logs path)"
CREATE TABLE [_llm_migrations] (
   [name] TEXT PRIMARY KEY,
   [applied_at] TEXT
);
CREATE TABLE "log" (
   [id] INTEGER PRIMARY KEY,
   [model] TEXT,
   [timestamp] TEXT,
   [prompt] TEXT,
   [system] TEXT,
   [response] TEXT,
   [chat_id] INTEGER REFERENCES [log]([id])
);

from llm.

There's other data about individual runs that I'm interested in storing. For non-streaming responses from OpenAI I get back this:

  "created": 1686896201,
  "id": "chatcmpl-7Rx3BL9grubSusAyCEiRoJta8vEh7",
  "model": "gpt-3.5-turbo-0301",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 399,
    "prompt_tokens": 15,
    "total_tokens": 414
  }
}

I don't think I get the "usage" block for streaming responses, which is annoying.

from llm.

I have another feature in the pipeline that will use a different model from the requested one:

• #32

That may want to store "user requested 'auto' but we ran gpt-4-32k." But that's evev more confusing, because there are actually three models there - auto was requested, gpt-4-32k was then selected, but gpt-4-32k-0601 or whatever was actually executed.

I think in that case I don't actually care that they said "auto".

from llm.

I'm going to add a duration_ms integer column to store the duration of the prompt, and a debug column which I'll dump JSON into with model-specific debug things - that's usage and model for the OpenAI ones and who-knows-what for the other models.

from llm.

@migration
def m005_debug(db):
    db["log"].add_column("debug", str)
    db["log"].add_column("duration_ms", int)

from llm.

Example output:

% llm logs
[
  {
    "id": 435,
    "model": "gpt-3.5-turbo",
    "timestamp": "2023-06-16 07:46:45.781006",
    "prompt": "say one duration",
    "system": null,
    "response": "1 hour",
    "chat_id": null,
    "debug": "{\"model\": \"gpt-3.5-turbo-0301\"}",
    "duration_ms": 820
  },
  {
    "id": 434,
    "model": "gpt-3.5-turbo",
    "timestamp": "2023-06-16 07:46:42.106479",
    "prompt": "say one duration",
    "system": null,
    "response": "One hour.",
    "chat_id": null,
    "debug": "{\"model\": \"gpt-3.5-turbo-0301\", \"usage\": {\"prompt_tokens\": 11, \"completion_tokens\": 3, \"total_tokens\": 14}}",
    "duration_ms": 1364
  },

from llm.

Updated schema: https://llm.datasette.io/en/latest/logging.html#sql-schema

from llm.