Be the first to get access to new connectors. Get early access →
ConnectorBay Early access

Langfuse evaluators, rules and annotation queues (your keys) — connector documentation

  • MCP endpoint: https://byok-langfuse-evaluators.connectorbay.com/mcp
  • REST API: POST https://byok-langfuse-evaluators.connectorbay.com/v1/tools/{tool_name}, described by the OpenAPI 3.1 spec at https://byok-langfuse-evaluators.connectorbay.com/openapi.json
  • Transport: Streamable HTTP, stateless, MCP specification 2025-06-18 or later
  • Authentication: your own Langfuse Evaluators Langfuse public key (starts with pk-lf-), sent in the X-Upstream-Key header and the Langfuse secret key (starts with sk-lf-) in X-Upstream-Key-2 of every request (never stored, logged or cached)
  • Health: https://byok-langfuse-evaluators.connectorbay.com/health · Tool schemas: https://byok-langfuse-evaluators.connectorbay.com/tools.json
  • Publisher: Unify Dynamics LLC · connectorbay.com/byok-langfuse-evaluators · agent@connectorbay.com

What Langfuse evaluators, rules and annotation queues (your keys) is for

Langfuse evaluators, rules and annotation queues (your keys) lets an AI assistant read your own Langfuse Evaluators data through the Langfuse Evaluators API with a key you supply (Langfuse public key (starts with pk-lf-)). Read-only: 10 tools (list_evaluators, get_evaluator, list_evaluator_versions, list_evaluation_rules, get_evaluation_rule, list_annotation_queues, get_annotation_queue, list_annotation_queue_items, list_dashboards, list_llm_connections). The BYOK key travels with each request, is used for that one call only and is never stored, logged or cached; it is only ever sent to cloud.langfuse.com. Unofficial connector by Unify Dynamics LLC: not affiliated with, endorsed by or sponsored by Langfuse Evaluators. Langfuse Evaluators and its logos are trademarks of their owners and are used only to say which service the connector works with.

Ask your assistant things like:

  • "Which evaluators are set up in my Langfuse project?"
  • "Show Langfuse evaluator ev_abc123."
  • "How many versions does Langfuse evaluator ev_abc123 have?"
  • "Which Langfuse evaluation rules are enabled and at what sampling rate?"
  • "Show Langfuse evaluation rule rule_1."
  • "What annotation queues exist in Langfuse?"

What it does

  • list_evaluators: List evaluators (LLM-as-judge and code) with name, type, status, current version and dates. The evaluator prompt and source code are not returned.
  • get_evaluator: Get one evaluator by id: name, type, status, pause reason, current version and dates. The evaluator prompt and source code are not returned.
  • list_evaluator_versions: List the versions of one evaluator, newest first, with version number and creation time. The evaluator prompt and source code are not returned.
  • list_evaluation_rules: List evaluation rules (which live data gets scored) with enabled flag, sampling rate and evaluator assignments.
  • get_evaluation_rule: Get one evaluation rule by id with enabled flag, sampling rate, filters and evaluator assignments.
  • list_annotation_queues: List annotation queues (human review queues) with name, description and the score configs used.
  • get_annotation_queue: Get one annotation queue by id: name, description and score config ids.
  • list_annotation_queue_items: List the items of an annotation queue (traces, observations or sessions to review) with status and completion time.
  • list_dashboards: List custom dashboards (unstable API) with name, description and dates; the widget definitions are not returned.
  • list_llm_connections: List the LLM provider connections configured for evaluators: provider, adapter and base URL (secrets are not returned).

BYOK key

You bring your own Langfuse Evaluators Langfuse public key (starts with pk-lf-) (https://cloud.langfuse.com). Your assistant sends it in the X-Upstream-Key header with each request. ConnectorBay uses it for that one call only: it is never stored, logged or cached, never shown in an answer, and it is only sent to cloud.langfuse.com over https, with no redirects followed. Create a key with the fewest permissions the tools need (read-only if the provider offers it), and revoke it at the provider whenever you like.

Data sources

Live from the Langfuse Evaluators API (cloud.langfuse.com) with a BYOK key; documentation: https://api.reference.langfuse.com/. Endpoints checked against that documentation on 2026-10-16.

What it is not

Unofficial connector by Unify Dynamics LLC: not affiliated with, endorsed by or sponsored by Langfuse Evaluators. Langfuse Evaluators and its logos are trademarks of their owners and are used only to say which service the connector works with. It is read-only: it never creates, changes, sends or deletes anything in your Langfuse Evaluators account.

Built for: Worldwide. Languages: English.