Confident AI Evals (BYOK)

Confident AI Evals (BYOK) — connector documentation

• MCP endpoint: https://byok-confident-ai-evals.connectorbay.com/mcp
• REST API: POST https://byok-confident-ai-evals.connectorbay.com/v1/tools/{tool_name}, described by the OpenAPI 3.1 spec at https://byok-confident-ai-evals.connectorbay.com/openapi.json
• Transport: Streamable HTTP, stateless, MCP specification 2025-06-18 or later
• Authentication: your own Confident AI Evals Confident AI project API key, sent in the X-Upstream-Key header of every request (never stored, logged or cached)
• Health: https://byok-confident-ai-evals.connectorbay.com/health · Tool schemas: https://byok-confident-ai-evals.connectorbay.com/tools.json
• Publisher: Unify Dynamics LLC · connectorbay.com/byok-confident-ai-evals · agent@connectorbay.com

What Confident AI Evals (BYOK) is for

Confident AI Evals (BYOK) lets an AI assistant read your own Confident AI Evals data through the Confident AI Evals API with a key you supply (Confident AI project API key). Read-only: 10 tools (list_metrics, get_metric, list_metric_versions, list_metric_collections, get_metric_collection, list_evaluation_rules, list_datasets, list_dataset_versions, list_personas, list_transformers). The BYOK key travels with each request, is used for that one call only and is never stored, logged or cached; it is only ever sent to api.confident-ai.com. Unofficial connector by Unify Dynamics LLC: not affiliated with, endorsed by or sponsored by Confident AI Evals. Confident AI Evals and its logos are trademarks of their owners and are used only to say which service the connector works with.

Ask your assistant things like:

• "Which custom evaluation metrics are defined in my Confident AI project?"
• "Show the definition of the Confident AI metric met_123"
• "What versions does the Confident AI metric met_123 have?"
• "Which metric collections does my Confident AI project have and what thresholds do they use?"
• "Show the Confident AI metric collection col_1"
• "Which automatic evaluation rules are running in my Confident AI project?"

What it does

• list_metrics: List the custom evaluation metrics of the project with name, algorithm, scoring criteria or steps, rubric and the test-case fields each metric needs.
• get_metric: Get one custom metric definition: algorithm, criteria or evaluation steps, rubric and required test-case fields.
• list_metric_versions: List the definition history of a custom metric, newest first, with version label, sequence number and creation time.
• list_metric_collections: List the metric collections with sampling rate and, per metric, threshold, strict mode, reason setting and the evaluation model used.
• get_metric_collection: Get one metric collection with every metric inside it and its settings.
• list_evaluation_rules: List the evaluation rules (name, enabled flag, whether they evaluate traces, spans or threads), newest first. Needs the Starter plan or above.
• list_datasets: List the datasets of the project, newest first, with id, alias and whether they are multi-turn (the goldens inside are not returned).
• list_dataset_versions: List the versions of one dataset, newest first, with version number and creation time. Needs the Team plan or above.
• list_personas: List the simulation personas (id and name only), newest first, used for multi-turn evaluation.
• list_transformers: List the payload transformers (name, description, dates) that metric collections and AI connections can use.

BYOK key

You bring your own Confident AI Evals Confident AI project API key (https://www.confident-ai.com/docs/settings/project/api-keys). Your assistant sends it in the X-Upstream-Key header with each request. ConnectorBay uses it for that one call only: it is never stored, logged or cached, never shown in an answer, and it is only sent to api.confident-ai.com over https, with no redirects followed. Create a key with the fewest permissions the tools need (read-only if the provider offers it), and revoke it at the provider whenever you like.

Notices

Reads evaluator and dataset configuration only (metric definitions, collections, rules, dataset aliases and version labels). Goldens, test cases, traces and annotations are not read. Uses the US region host; EU-region projects use a different host that this connector does not call.

Data sources

• https://www.confident-ai.com/docs/reference/api/v2/metrics/list-metrics (the API this connector calls, with a BYOK key)

Live from the Confident AI Evals API (api.confident-ai.com) with a BYOK key; documentation: https://www.confident-ai.com/docs/reference/api/v2/metrics/list-metrics. Endpoints checked against that documentation on 2026-10-16.

What it is not

Unofficial connector by Unify Dynamics LLC: not affiliated with, endorsed by or sponsored by Confident AI Evals. Confident AI Evals and its logos are trademarks of their owners and are used only to say which service the connector works with. It is read-only: it never creates, changes, sends or deletes anything in your Confident AI Evals account.

Built for: Worldwide. Languages: English.

Confident AI Evals (BYOK) — Privacy Policy

Effective 24 October 2026. Unify Dynamics LLC ("we", "us"), Menlo Park, California.

What Confident AI Evals (BYOK) is. Confident AI Evals (BYOK) is a connector (an MCP server and REST API) that an AI assistant calls on your behalf to read your own Confident AI Evals data: Read the custom metrics, metric collections, evaluation rules, dataset names and versions of your Confident AI project with your own project API key. You bring your own Confident AI Evals Confident AI project API key; the assistant sends it to us with each request.

BYOK key. The key you supply is used to make the one Confident AI Evals call you asked for. Our application does not store it, write it to a log, put it in a cache, or share it. It is sent only to api.confident-ai.com, over https, and is removed from any text we return to you or your assistant. We do not follow redirects with your key. If Confident AI Evals rejects it, we say so without repeating it.

What we receive. The arguments of each tool call, the BYOK key (for that call only), the technical headers needed to serve the request (an IP address or a client identifier supplied by the assistant) and standard server logs. What Confident AI Evals receives. The request we make on your behalf, including the key and the arguments, under Confident AI Evals's own terms and privacy policy. Confident AI Evals acts at your direction and is the only recipient of the data of your request besides our hosting provider.

What we return. The answer Confident AI Evals gives for your request, trimmed to the fields the tool documents. We do not keep it: it is returned to your assistant and not stored, cached or used to train anything.

What we do not do. We do not build a profile, sell or share personal information, or keep your Confident AI Evals data. Rate-limit counters (a client identifier and timestamps) are held in memory for at most 24 hours. Hosting access logs (IP addresses, request paths and status codes) are kept for up to 30 days for security and abuse prevention; our application adds no request headers or bodies to them.

Sub-processors. Confident AI Evals (the API you call with a BYOK key, at your direction). Hosting: Vercel (United States regions).

Your rights. Because we store no personal data beyond short-lived logs, requests to access, correct or delete data are usually satisfied automatically; you can still contact agent@connectorbay.com and we will respond within 30 days. Revoke your key at Confident AI Evals at any time to end all access.

Children. Confident AI Evals (BYOK) is a general-audience service operated by the adult using the assistant, under Confident AI Evals's own age rules.

Changes. We will post changes here with a new effective date.

Confident AI Evals (BYOK) — Terms of Service

Effective 24 October 2026. These terms govern use of the Confident AI Evals (BYOK) connector ("the Service") operated by Unify Dynamics LLC ("we", "us").

1. The Service. Confident AI Evals (BYOK) provides: Read the custom metrics, metric collections, evaluation rules, dataset names and versions of your Confident AI project with your own project API key. It is accessed through an AI assistant or directly over MCP or HTTP, and is read-only.
2. Your own key, your own account. You must use your own Confident AI Evals credentials and only access data you are allowed to access. Confident AI Evals's terms apply to your use of Confident AI Evals; you are responsible for complying with them, including any rate limits and permitted uses. Do not share a key you do not own.
3. Independent. Unofficial connector by Unify Dynamics LLC: not affiliated with, endorsed by or sponsored by Confident AI Evals. Confident AI Evals and its logos are trademarks of their owners and are used only to say which service the connector works with.
4. No advice, no guarantees. Output is the data Confident AI Evals returns, lightly trimmed; it may be incomplete, delayed or changed by Confident AI Evals. Check important facts at the source.
5. Read-only. The Service does not create, change, send or delete anything in your Confident AI Evals account.
6. Acceptable use. Automated use within the published rate limits is welcome. Do not use the Service to access data you are not entitled to, to circumvent Confident AI Evals's controls, or to attack any system.
7. Availability. The Service is provided "as is" without warranty. We may change, limit or discontinue it at any time; Confident AI Evals may change or end its API at any time.
8. Data. Our Privacy Policy describes what we process. In short: the BYOK key is used for the one call and not stored or logged.
9. Pro keys. Optional ConnectorBay Pro API keys are issued at our discretion and are separate from your Confident AI Evals key.
10. Liability. To the fullest extent permitted by law, Unify Dynamics LLC is not liable for any loss arising from use of or reliance on the Service, including any action Confident AI Evals takes on your account. Our total liability is limited to US $100.
11. Governing law. California law; venue in San Mateo County, California.
12. Contact. Unify Dynamics LLC · agent@connectorbay.com · https://connectorbay.com/byok-confident-ai-evals

Confident AI Evals (BYOK) support

Email agent@connectorbay.com. We answer within two business days.

• MCP endpoint: https://byok-confident-ai-evals.connectorbay.com/mcp
• REST API: https://byok-confident-ai-evals.connectorbay.com/openapi.json
• Live status: https://byok-confident-ai-evals.connectorbay.com/health

When you report a problem, include the question you asked your agent, roughly when, and which agent you used (for example Claude, ChatGPT or Gemini). Never send us passwords, account numbers or ID numbers. Confident AI Evals (BYOK) never asks for them.