langfuse / langfuse Goto Github PK

Describe the bug

In pagination, when we change the 'rows per page' option, ideally the page number should change, but it's not happening

page number should increase

To reproduce

try to change the "rows per page" in traces pagination

Additional information

No response

_LFC-167

API and UI reference it as publicKey; in database and application still publishableKey

_{From SyncLinear.com | LF-177}

Describe the feature you'd like to request

I would like to self host langfuse in the cloud on a public URL and limit who can log in to langfuse.
Currently if the system/URL is public, anybody could sign up and log in.

Describe the solution you'd like to see

I'm not sure what a good solution would be for this.

An example solution I can think off:

• having a setting to disable sign up

Then first I would deploy while sign up is enabled and create an account.
Then I disable sign up and deploy again

Usage:

docker run --name langfuse \
-e DATABASE_URL=...
...
SIGN_UP_DISABLED=TRUE
ghcr.io/langfuse/langfuse:latest

Additional information

No response

Contribute

• Yes, I can implement this and raise a PR

Issue is related to the Python SDK: https://github.com/langfuse/langfuse-python

For users who just want to log OpenAI calls to Langfuse, getting started with Langfuse could be as simple as:

# openai
from openai import openai

# openai with langfuse logging
from langfuse.openai import openai

_{From SyncLinear.com | LFC-179}

E.g. using https://github.com/mac-s-g/react-json-view

_LFC-210

Describe the feature you'd like to request

Add delete project button to project settings

Describe the solution you'd like to see

• dialog
  • use shadcn ui button component
  • modal to confirm deletion
• trpc route for deletion
• project:delete as new scope only granted to project owners
  • hide button in frontend if not owner
  • enforce in trpc route that user needs to be owner to delete project
• cascading deletion of observation/traces/scores/memberships in db

Additional information

No response

Contribute

• Yes, I can implement this and raise a PR

_{From SyncLinear.com | LFC-178}

Expected behavior

• Support pagination in the UI to go to the next page of a table
• Expand the trpc API to (1) fetch total number of rows, (2) paginate

Status

• Update component: #118
• Add pagination for
  • Traces: #118
  • Observations
  • Scores
  • Users

Reference implementation

Already implemented for trace table: #118

_{From SyncLinear.com | LF-148}

Tracking issue for:

• https://github.com/langfuse/langfuse/security/code-scanning/1

_LFC-216

Goal: simplify it to jump from one trace/generation to the next in a list

Example (rough sketch)

Nice to have: keyboard navigation (k & j)

Thoughts on potential implementation

• Challenge: how to store state of (filtered) table to actually page through items of the previously visited table view
  • store list & link to table in
    • session storage
    • query param

_LFC-215

Describe the feature you'd like to request

Add CONTRIBUTING.md File

Describe the solution you'd like to see

CONTRIBUTING.md File , if will helpful for new contributor

Additional information

No response

Similar to implementation of traces table

Search (contains)

• generationId
• name

_LFC-207

Describe the feature you'd like to request

Currently users can only be created and the initial password that is chosen at /auth/sign-up is hashed and saved to user table. Users cannot change or reset the password.

Suggested core feature

• New page for user settings accessible via the user menu that currently only includes "sign out"
• Form to set a new password while being logged in

Users who do not know there current password, cannot change it -> need password reset flow; this would require adding transactional emails which is probably out of scope to make this easy to finish. Happy to contribute if you want to go for it though.

Describe the solution you'd like to see

• Add update route to user trpc router, user id of signed in user is available in context (ctx.session.user.id) -> src/server/api/routers/users.ts
  • Update user table using prisma

Additional information

The project uses:

• shadcn/ui for ui components, find them in src/components/ui
• trpc for typed APIs used by the frontend, check out how the creation of new API keys works for reference
  • src/features/publicApi/components/CreateApiKeyButton.tsx
  • src/features/publicApi/server/apiKeyRouter.ts

_{From SyncLinear.com | LFC-6}

_LFC-85

Currently, we are able to generate Langfuse callback handlers based on traces. See docs here.

I think it would be great to have the same for observations. We have users, who want to call Langchain multiple times per trace and hence we need a way to wrap Langchain calls into spans. Right now, in the callback handler, we always create a Trace if none exist, and spans and generations based on the hierarchy given by Langchain via run_id and parent_run_id For this, we would need to:

• be able to instantiate the Langchain Callback handler with an observationId
• In all places, where we usually generate a Trace and Span (root of a Langchain execution), instead of generating a Trace, only generate the span with the observationId as parentObservationId

_LFC-181

Describe the feature you'd like to request

I suggest having a proper template for PRs. When a person contributes and makes a PR if the information needs to be filled like a description of the feat/bug, which issue it resolves etc.

Describe the solution you'd like to see

A template for PR like :

Description
Describe what changes you made.

Issue Resolved # (issue)

Checks

• My code passes all tests
• I have formatted the code
• It generates no new warnings
• Have written documentation or added comments for it

Additional information

No response

Contribute

• Yes, I can implement this and raise a PR

Describe the feature you'd like to request

Would love to be able to set metadata fields on manually created spans:

Describe the solution you'd like to see

langfuse_handler.setNextSpan(next_span_id, metadata_field_1="foo",metadata_field_2="bar")

Additional information

No response

Contribute

• Yes, I can implement this and raise a PR

Describe the feature you'd like to request

Currently, the Create* models, e.g. CreateGeneration declare all arguments as required to the static type checking. This is unhelpful, because most of them accept None as the default value and work fine.

I think the core confusion is around typing.Optional used in the models. It means the field can be set to None, but the field is still considered required, unless there is a default provided.

Describe the solution you'd like to see

Here's a minimal example to consider. The current code looks like this:

from pydantic import BaseModel
from typing import Optional
class Model(BaseModel):
   field:  Optional[str]

r = Model()  # error, the field is required
r = Model(field=None)  # OK
r = Model(field="fool)  # OK

It could be changed to this instead:

class Model(BaseModel):
    field: Opitional[str] = None

r = Model() # OK
r = Model(field=None)  # OK, same result

Additional information

No response

Contribute

• Yes, I can implement this and raise a PR

In the Python package, users can create Langchain handlers based on a trace. Here are the docs.

The same behavior does not exist for this JS/TS package yet, but we want to change that. The core langfuse SDK is built in a way to support es5. Langchain does not support es5, hence we have to build it into the langchain-langfuse package.

I would take the following approach:

• Wrap the Langchain SDK (LangfuseCore and LangfuseTraceClient) in the langchain-langfuse package
• Add a method to LangfuseTraceClient to export the callback handler based on a trace
• For LangfuseCore::trace, return the LangfuseTraceClient
• Change the callback handler to take a traceId. Within the callback handler, before creating a trace, ensure that the traceId is not set. Otherwise, use the traceId to generate spans and generations

_LFC-185

Currently, there is an analytics alpha based on Looker. The next step is to replace it with a full open source analytics stack.

Goals

• Fast
• Fully OSS
• Self-serve chart builder
• Support for some metrics/semantic layer of prebuilt aggregations
• Prebuilt dashboards

_LFC-193

Describe the bug

After starting self-hosted Langfuse, interaction with the signup/login mechanism hangs. Usually a refresh of the page will resolve the issue.

To reproduce

Here's my docker compose file:

version: '3.8'
services:
  <other-service-A>:
    ...
  <other-service-B>:
    ...
  postgres:
    image: postgres:latest
    environment:
      POSTGRES_USER: xxxx
      POSTGRES_PASSWORD: xxxx
      POSTGRES_DB: xxxx
    ports:
      - "5432:5432"
    volumes:
      - ${HOME}/data/postgres:/var/lib/postgresql/data
    networks:
      - network1
  langfuse:
    image: ghcr.io/langfuse/langfuse:latest
    environment:
      DATABASE_URL: postgresql://xxxx:xxxx@postgres:5432/postgres
      NEXTAUTH_SECRET: mysecret
      NEXTAUTH_URL: http://XXX.XX.0.1:3002
    ports:
      - 3002:3000
    networks:
      - network1
networks:
  network1:
    name: my-network
    attachable: true
    ipam:
      driver: default
      config:
        - subnet: XXX.XX.0.0/16
          ip_range: XXX.XX.5.0/24
          gateway: XXX.XX.0.1

Additional information

Network

_LFC-195

Callback handler for haystack
https://github.com/deepset-ai/haystack

Chainlit integration with haystack might be a good reference implementation
https://docs.chainlit.io/integrations/haystack

_LFC-196

We should have a simple Kubernetes setup to get running fast. We would need:

• Kubernetes deployment
• Kubernetes service
• Secret configs
• DB config via env variables

Optional: docs, can be follow-up issue

_{From SyncLinear.com | LFC-203}

Pinned post with collection of large roadmap items

• Integrations
  • #188
  • #192
  • #187
  • #199
  • #227
• New capabilities
  • #196
  • #197
  • #200
  • #194

_{From SyncLinear.com | LFC-191}

Users want to be able to update their traces as they are able to update spans and generations. The user should have the following experience:

trace = langfuse.trace(CreateTrace(...))
trace = trace.update(UpdateTrace(...))

# ability to generate everything that can be a child of a trace
span = trace.span(...)
generation = trace.generation(...)

• Create a StatefulTraceClient which is a StatefulClient but is also able to update Traces.
  • To update the Traces, we can use the same request as when creating the Trace, as the API route upserts.
  • Langfuse::trace should return the StatefulTraceClient
  • Langfuse::score should return the StatefulTraceClient in case we scored a Trace
• Add an integration test for the added functionality
• Add docs to the docs repo (https://github.com/langfuse/langfuse-docs)

_LFC-183

Currently, traces can be shared publicly via /public/traces/:traceid

While the traces are intended to be public when shared, the ids can be guessed (if they are incremental) or brute forced (if they are short). Trace ids default to uuids but can be set manually via the API/SDKs leading to potentially short/sequential ids.

Example: https://cloud.langfuse.com/public/traces/lf.docs.conversation.4AO8dBC

Desired outcome

Before: /public/traces/my-guessable-id-1

After: /public/traces/my-guessable-id-1?token=7v8dnt782itbguidt7t7c6ds9t

Suggested implementation

• New column publicToken in table traces (via Prisma ORM)
• Create new random token whenever trace is shared via trpc route; delete when unshared
• Add token to url of public link as queryparam (?token=…)
  • Add it to url that is copied when creating the sharable link
  • Check if token is correct in byIdPublic trpc route on /public/traces/:traceid page

_{From SyncLinear.com | LFC-164}

Many users have problems with OpenAI rate limiting and OpenAI provides response headers to track current usage of the API.

Details: https://platform.openai.com/docs/guides/rate-limits/rate-limits-in-headers

Suggested change

• Capture the known response headers for OpenAI generations in the Langchain Callback handler (on_llm_end)
• Add them as key-value pairs to the metadata of the generation in Langfuse Update (UpdateGeneration)

Happy to support on this, ping me on discord!

_LFC-182

Filter tables of generations and export selected rows or all as json, csv

Steps

• Select, more filters
• Export files: csv, json, jsonl (openai)

_{From SyncLinear.com | LFC-186}

Describe the feature you'd like to request

Currently users can only be created and the initial password that is chosen at /auth/sign-up is hashed and saved to user table. Users cannot change or reset the password.

Suggested core feature

• New page for user settings accessible via the user menu that currently only includes "sign out"
• Form to set a new password while being logged in

Users who do not know there current password, cannot change it -> need password reset flow; this would require adding transactional emails which is probably out of scope to make this easy to finish. Happy to contribute if you want to go for it though.

Describe the solution you'd like to see

• Add update route to user trpc router, user id of signed in user is available in context (ctx.session.user.id) -> src/server/api/routers/users.ts
  • Update user table using prisma

Additional information

The project uses:

• shadcn/ui for ui components, find them in src/components/ui
• trpc for typed APIs used by the frontend, check out how the creation of new API keys works for reference
  • src/features/publicApi/components/CreateApiKeyButton.tsx
  • src/features/publicApi/server/apiKeyRouter.ts

_LF-655

Goal: make sure only dependencies are added to the project which have a public domain or permissive license, no copyleft.

Looks like a good starting point: https://github.com/marketplace/actions/dependencies-license-compliance-checker

_LFC-209

The Q&A chatbot does not illustrate the user-based analytics feature since users are anonymous while using the chatbot. One solution to be able to show the user-based analytics with the q&a chatbot (it's the public demo) is to use a random id stored in SessonStorage to identify a user

_LFC-204

We should start testing the frontend application as well to control for regressions affecting the front-end application.

To get started, I'd suggest to add playwright testing to the tests run on CI (npm run test). Ideas for test cases:

• Init: #242
• Create account #390
• Sign in: #242
• Create new project: #392

_{From SyncLinear.com | LFC-202}

For users who just want to log OpenAI calls to Langfuse, getting started with Langfuse could be as simple as:

// openai
import OpenAI from 'openai';

const openai = new OpenAI({
  apiKey: 'my api key',
});

// openai with Langfuse
import OpenAI from 'langfuse-openai';

const openai = new OpenAI({
  apiKey: 'my api key',
  langfuseSecretKey: 'lf-sk-...',
  langfusePublicKey: 'lf-pk-...',
});

_LFC-184

• Create a react hook which recognises keyboard input
• provide an interace for configuration: view where the action should be placed, key, function to be executed once keyboard is hit

_{From SyncLinear.com | LFC-206}

OTel as (1) additional data source for more execution details, and (2) alternative integration to replace current SDKs

_LFC-192

Describe the bug

To reproduce

Run langfuse container on Google Cloud Run

Additional information

Optional:

• Add cloud run instructions to docs

_LFC-188

Describe the bug

When click Analytics on the live demo, the Usage, Latency, Scores tab always reporting Please sign in to your Google account, then reload this page to view this report, but I have alraedy logged in to my Google account, am I missing anything?

To reproduce

• Login to Langfuse
• navigate to langfuse-docs project
• click Analytics, you will see the message

Additional information

No response

API and UI reference it as publicKey; in database and application still publishableKey

Current behavior

When clicking on specific Generations or Spans in the detail view of traces, we adjust the link. The Link has the following structure: https://cloud.langfuse.com/project/<project-id>/traces/<trace-id>?observation=<observation-id>

The changes according to the navigation in the Trace.

When opening this Link, the Trace Detail view renders with the trace navigation scrolled all the way up. In the following picture, we cannot see the currently opened Generation, which should have a grey background.

Desired behavior

Traces can become very long and finding the specific Generation or Span can be tedious. Hence it would be great to scroll to the right element on page load as in the following picture

_{From SyncLinear.com | LFC-171}

Create new documentation pages in https://github.com/langfuse/langfuse-docs for these new integrations

• LiteLLM: https://docs.litellm.ai/docs/observability/langfuse_integration -> /docs/integrations/litellm
  • langfuse/langfuse-docs#66
• Langflow -> /docs/integrations/langflow
  • https://docs.langflow.org/guides/langfuse_integration
  • langfuse/langfuse-docs#74
  • langfuse/langfuse-docs#79
• Flowise -> /docs/integrations/flowise
  • https://github.com/FlowiseAI/Flowise
  • langfuse/langfuse-docs#88

_{From SyncLinear.com | LFC-194}

Expected behavior

• Support pagination in the UI to go to the next page of a table
• Expand the trpc API to (1) fetch total number of rows, (2) paginate

Status

• Update component: #118
• Add pagination for
  • Traces: #118
  • Observations: #230
  • Scores: #231
  • Users: #231

Reference implementation

Already implemented for trace table: #118

_{From SyncLinear.com | LFC-160}

• Manage set of reference prompts/inputs to run tests on
• Add to the dataset from production traces
• Use within SDKs to run experiments

Workflows from SDK

1. Get all data set items by data set name (input & expected output), all that are not archived
2. User creates run (name) on data set
3. User runs app on each item and traces it with langfuse; needs to provide data set item id and run id (can both be from sdk state maybe) that the execution trace belongs to
4. Optionally: user runs evals and adds scores to the observation or trace

Todo

• Average score on run

_{From SyncLinear.com | LFC-189}

Most of the pages does not have any title so it showing the route as default and which look odd.

Current behavior

• Our package requires Pydantic version < 2.0.0. We have some who would like to run it with > 2.0.0.
• Fern has an issue on how to make changes in their code to support both versions: https://github.com/fern-api/fern-python/issues/288

Expected behaviror

• Make our SDK agnostic to Pydantic major versions
• Find a way with e.g. poetry dependency groups to test the different environments

_{From SyncLinear.com | LFC-199}

Provide environment information

[email protected]
Ok to proceed? (y)

System:
OS: Windows 10 10.0.22621
CPU: (8) x64 AMD Ryzen 7 3700U with Radeon Vega Mobile Gfx
Memory: 3.95 GB / 13.95 GB
Binaries:
Node: 18.13.0 - C:\Program Files\nodejs\node.EXE
Yarn: 1.22.19 - ~\AppData\Roaming\npm\yarn.CMD
npm: 9.6.7 - C:\Program Files\nodejs\npm.CMD
pnpm: 7.15.0 - ~\AppData\Roaming\npm\pnpm.CMD

Describe the bug

Link to reproduction

https://cloud.langfuse.com/project/clkpwwm0m000gmm094odg11gi/analytics

To reproduce

check the above link

Additional information

No response

• Pick and choose from catalog of evals
• Specify on which trace/generation types the evals shall run
• Async worker executes evals, also on historic production runs (not only at ingestion time)

_LFC-190

Describe the feature you'd like to request

I'm trying to make a callback to send data to langfuse. C
This will allow me to use langfuse with other tools like promptlayer, helicone etc

Describe the solution you'd like to see

Can you expose an API endpoint that allows me to pass langfuse the ChatCompletion() kwargs, response etc ?

Additional information

No response

_LF-657

Add docs for #194

• Reference functions in Python SDK as alternative route to pull data from Langfuse
• Video

_LFC-200

Why

• Good for docs / marketing
• Helps people share with coworkers

How

• public flag on trace
• can be set via button in frontend
• New /public/traces/:traceId page
• Alternative tRPC router to get the trace when not being authenticated as long as the trace is public

_{From SyncLinear.com | LFC-162}

Learn more: https://gpt-index.readthedocs.io/en/latest/end_to_end_tutorials/one_click_observability.html

Implementation similar to existing Langchain callback handler that we have built on top of the Python SDK

_{From SyncLinear.com | LFC-180}

Currently the Python and JS/TS SDK expect a private and secret key in their respective constructor. It would be nice to set these via environment variables, e.g. for use in staging and CI:

Envs

• LANGFUSE_PUBLIC_KEY, in JS also check NEXT_PUBLIC_LANGFUSE_PUBLIC_KEY as there are many users using Next.JS for their applications
• LANGFUSE_SECRET_KEY
• LANGFUSE_HOST

This will lead to not requiring these params in the constructors. Log info to console that Langfuse is disabled if public key or secret key are missing.

Edit: envs should be an alternative way to set public/secret key and host. We need to support constructor arguments for backwards compatibility. Thus, they become optional.

• Python SDK
• JS SDK

_{From SyncLinear.com | LFC-201}