LangSmith Evaluation Config (BYOK)

LangSmith Evaluation Config (BYOK) — connector documentation

• MCP endpoint: https://byok-langsmith-evaluation-config.connectorbay.com/mcp
• REST API: POST https://byok-langsmith-evaluation-config.connectorbay.com/v1/tools/{tool_name}, described by the OpenAPI 3.1 spec at https://byok-langsmith-evaluation-config.connectorbay.com/openapi.json
• Transport: Streamable HTTP, stateless, MCP specification 2025-06-18 or later
• Authentication: your own LangSmith Evaluation Config LangSmith API key (personal access token or service key), sent in the X-Upstream-Key header of every request (never stored, logged or cached)
• Health: https://byok-langsmith-evaluation-config.connectorbay.com/health · Tool schemas: https://byok-langsmith-evaluation-config.connectorbay.com/tools.json
• Publisher: Unify Dynamics LLC · connectorbay.com/byok-langsmith-evaluation-config · agent@connectorbay.com

What LangSmith Evaluation Config (BYOK) is for

LangSmith Evaluation Config (BYOK) lets an AI assistant read your own LangSmith Evaluation Config data through the LangSmith Evaluation Config API with a key you supply (LangSmith API key (personal access token or service key)). Read-only: 6 tools (list_evaluators, get_evaluator, get_evaluator_spend, list_feedback_configs, list_feedback_formulas, list_run_rules). The BYOK key travels with each request, is used for that one call only and is never stored, logged or cached; it is only ever sent to api.smith.langchain.com. Unofficial connector by Unify Dynamics LLC: not affiliated with, endorsed by or sponsored by LangSmith Evaluation Config. LangSmith Evaluation Config and its logos are trademarks of their owners and are used only to say which service the connector works with.

Ask your assistant things like:

• "Which evaluators are defined in my LangSmith workspace?"
• "Show the LangSmith evaluator 3f1c2a9e-8b7d-4c1e-9a55-0d6f2b7a1c10"
• "How much did my LangSmith evaluators cost since the start of this month?"
• "Which feedback keys and score ranges are configured in LangSmith?"
• "Which composite feedback formulas exist in LangSmith?"
• "Which automatic evaluation rules run in my LangSmith workspace?"

What it does

• list_evaluators: List the evaluators of the workspace (LLM-as-judge and code evaluators) with feedback keys, managed flag, dates and their definition.
• get_evaluator: Get one evaluator: type, feedback keys, LLM judge settings (prompt repository, commit or tag, few-shot count) or code settings, and the run rules that use it.
• get_evaluator_spend: Get evaluator spend in USD and number of traces evaluated for a period, grouped by evaluator and rule, with the previous period for comparison.
• list_feedback_configs: List the feedback key schemas (continuous range, categorical categories or freeform) with whether a lower score is better and whether the schema is enforced.
• list_feedback_formulas: List the composite feedback formulas (aggregation type and weighted parts) defined for datasets or projects.
• list_run_rules: List the online evaluation and automation rules (sampling rate, filters, evaluator, target dataset or annotation queue, spend limit, trace count and spend). Alert and webhook settings are not returned.

BYOK key

You bring your own LangSmith Evaluation Config LangSmith API key (personal access token or service key) (https://smith.langchain.com/settings). Your assistant sends it in the X-Upstream-Key header with each request. ConnectorBay uses it for that one call only: it is never stored, logged or cached, never shown in an answer, and it is only sent to api.smith.langchain.com over https, with no redirects followed. Create a key with the fewest permissions the tools need (read-only if the provider offers it), and revoke it at the provider whenever you like.

Notices

Reads evaluation setup only: evaluator and rule definitions, feedback key schemas, formulas and spend totals. Runs, traces, feedback scores, examples and alert or webhook settings of rules are not returned. Uses the US cloud host; EU-region workspaces use a different host that this connector does not call.

Data sources

• https://docs.langchain.com/langsmith/langsmith-platform-openapi.json (the API this connector calls, with a BYOK key)

Live from the LangSmith Evaluation Config API (api.smith.langchain.com) with a BYOK key; documentation: https://docs.langchain.com/langsmith/langsmith-platform-openapi.json. Endpoints checked against that documentation on 2026-10-16.

What it is not

Unofficial connector by Unify Dynamics LLC: not affiliated with, endorsed by or sponsored by LangSmith Evaluation Config. LangSmith Evaluation Config and its logos are trademarks of their owners and are used only to say which service the connector works with. It is read-only: it never creates, changes, sends or deletes anything in your LangSmith Evaluation Config account.

Built for: Worldwide. Languages: English.

LangSmith Evaluation Config (BYOK) — Privacy Policy

Effective 24 October 2026. Unify Dynamics LLC ("we", "us"), Menlo Park, California.

What LangSmith Evaluation Config (BYOK) is. LangSmith Evaluation Config (BYOK) is a connector (an MCP server and REST API) that an AI assistant calls on your behalf to read your own LangSmith Evaluation Config data: Read the evaluators, online evaluation rules, feedback label schemas, scoring formulas and evaluator spend of your LangSmith workspace with your own API key. You bring your own LangSmith Evaluation Config LangSmith API key (personal access token or service key); the assistant sends it to us with each request.

BYOK key. The key you supply is used to make the one LangSmith Evaluation Config call you asked for. Our application does not store it, write it to a log, put it in a cache, or share it. It is sent only to api.smith.langchain.com, over https, and is removed from any text we return to you or your assistant. We do not follow redirects with your key. If LangSmith Evaluation Config rejects it, we say so without repeating it.

What we receive. The arguments of each tool call, the BYOK key (for that call only), the technical headers needed to serve the request (an IP address or a client identifier supplied by the assistant) and standard server logs. What LangSmith Evaluation Config receives. The request we make on your behalf, including the key and the arguments, under LangSmith Evaluation Config's own terms and privacy policy. LangSmith Evaluation Config acts at your direction and is the only recipient of the data of your request besides our hosting provider.

What we return. The answer LangSmith Evaluation Config gives for your request, trimmed to the fields the tool documents. We do not keep it: it is returned to your assistant and not stored, cached or used to train anything.

What we do not do. We do not build a profile, sell or share personal information, or keep your LangSmith Evaluation Config data. Rate-limit counters (a client identifier and timestamps) are held in memory for at most 24 hours. Hosting access logs (IP addresses, request paths and status codes) are kept for up to 30 days for security and abuse prevention; our application adds no request headers or bodies to them.

Sub-processors. LangSmith Evaluation Config (the API you call with a BYOK key, at your direction). Hosting: Vercel (United States regions).

Your rights. Because we store no personal data beyond short-lived logs, requests to access, correct or delete data are usually satisfied automatically; you can still contact agent@connectorbay.com and we will respond within 30 days. Revoke your key at LangSmith Evaluation Config at any time to end all access.

Children. LangSmith Evaluation Config (BYOK) is a general-audience service operated by the adult using the assistant, under LangSmith Evaluation Config's own age rules.

Changes. We will post changes here with a new effective date.

LangSmith Evaluation Config (BYOK) — Terms of Service

Effective 24 October 2026. These terms govern use of the LangSmith Evaluation Config (BYOK) connector ("the Service") operated by Unify Dynamics LLC ("we", "us").

1. The Service. LangSmith Evaluation Config (BYOK) provides: Read the evaluators, online evaluation rules, feedback label schemas, scoring formulas and evaluator spend of your LangSmith workspace with your own API key. It is accessed through an AI assistant or directly over MCP or HTTP, and is read-only.
2. Your own key, your own account. You must use your own LangSmith Evaluation Config credentials and only access data you are allowed to access. LangSmith Evaluation Config's terms apply to your use of LangSmith Evaluation Config; you are responsible for complying with them, including any rate limits and permitted uses. Do not share a key you do not own.
3. Independent. Unofficial connector by Unify Dynamics LLC: not affiliated with, endorsed by or sponsored by LangSmith Evaluation Config. LangSmith Evaluation Config and its logos are trademarks of their owners and are used only to say which service the connector works with.
4. No advice, no guarantees. Output is the data LangSmith Evaluation Config returns, lightly trimmed; it may be incomplete, delayed or changed by LangSmith Evaluation Config. Check important facts at the source.
5. Read-only. The Service does not create, change, send or delete anything in your LangSmith Evaluation Config account.
6. Acceptable use. Automated use within the published rate limits is welcome. Do not use the Service to access data you are not entitled to, to circumvent LangSmith Evaluation Config's controls, or to attack any system.
7. Availability. The Service is provided "as is" without warranty. We may change, limit or discontinue it at any time; LangSmith Evaluation Config may change or end its API at any time.
8. Data. Our Privacy Policy describes what we process. In short: the BYOK key is used for the one call and not stored or logged.
9. Pro keys. Optional ConnectorBay Pro API keys are issued at our discretion and are separate from your LangSmith Evaluation Config key.
10. Liability. To the fullest extent permitted by law, Unify Dynamics LLC is not liable for any loss arising from use of or reliance on the Service, including any action LangSmith Evaluation Config takes on your account. Our total liability is limited to US $100.
11. Governing law. California law; venue in San Mateo County, California.
12. Contact. Unify Dynamics LLC · agent@connectorbay.com · https://connectorbay.com/byok-langsmith-evaluation-config

LangSmith Evaluation Config (BYOK) support

Email agent@connectorbay.com. We answer within two business days.

• MCP endpoint: https://byok-langsmith-evaluation-config.connectorbay.com/mcp
• REST API: https://byok-langsmith-evaluation-config.connectorbay.com/openapi.json
• Live status: https://byok-langsmith-evaluation-config.connectorbay.com/health

When you report a problem, include the question you asked your agent, roughly when, and which agent you used (for example Claude, ChatGPT or Gemini). Never send us passwords, account numbers or ID numbers. LangSmith Evaluation Config (BYOK) never asks for them.