AI & Machine Learning · Scorecard MCP + Claude
Scorecard MCP with Claude: Run and Track AI Evaluations Through Natural Language
The short answer
Claude works with Scorecard through the Model Context Protocol (MCP). Once connected, Claude can read evaluation projects, testsets, metrics and runs, record scores and annotations, and search documentation, all through natural language. Permissions control what it can change, and people approve important actions.
What is Scorecard MCP?
Scorecard is an evaluation platform for AI teams. It keeps testsets, metrics and evaluation runs in one place, so teams can measure how well their AI models, agents and chatbots perform. Typical users include machine learning engineers, AI product managers, quality leads and data teams who need a consistent record of how each model version behaves before and after release.
How Claude connects to Scorecard MCP through MCP
The Model Context Protocol, or MCP, is an open standard that lets Claude use other software securely. Think of it as a universal adapter between Claude and a tool. The organisation decides which actions Claude is allowed to take in that tool, and access is limited to what each task needs.
Once Scorecard MCP is connected, Claude can typically look up evaluation projects, browse testsets, review metrics and past runs, record scores and annotations, and search the platform's documentation. A team member can ask a question such as which runs failed a given metric last week, and Claude can gather the answer from the platform.
Permissions and approvals keep people in charge. Access uses Dynamic Client Registration OAuth sign-in, so each connection is authorised through the normal sign-in process rather than shared passwords. Actions that change records, such as starting a run or updating scores, can be set to require a person's approval before they take effect.
What you can automate
Evaluation run summaries for release reviews
Before a model release, a product manager asks Claude to summarise the latest evaluation runs for a chosen project. Claude pulls the relevant runs, compares key metrics against the previous version and drafts a short summary. The team reviews the draft and uses it in the release meeting, saving time spent hunting through dashboards.
Testset maintenance and coverage checks
A quality lead asks Claude which testsets have not been updated in several months or which product areas have few test cases. Claude reviews the testsets and reports gaps. The team then decides which new cases to add, keeping evaluations relevant as the product changes.
Annotation and scoring support for reviewers
Human reviewers often score AI outputs one by one. Claude can help by suggesting annotations based on agreed criteria, which reviewers accept, edit or reject. Recorded scores stay under human control, and the audit trail shows who made each decision.
Documentation search for new team members
A new engineer asks how to set up a particular metric or run type. Claude searches the Scorecard documentation and gives a clear answer with the relevant steps. This shortens onboarding and reduces repeated questions to senior staff.
Weekly quality reporting
An AI operations manager asks Claude each Monday to collect the week's evaluation results across several projects and list any metrics that dropped. Claude produces a consistent report that the team shares with leadership, reducing manual compilation work.
An end-to-end workflow: Chatbot release quality gate
TriggerA developer marks a new chatbot version ready for testing in the team's project tracker.
Claude reads the ticket in the project tracker and identifies the chatbot version and target evaluation project.
Claude uses Scorecard MCP to select the approved testset and start an evaluation run, subject to a reviewer's approval.
When the run completes, Claude compares each metric against the agreed thresholds and records the scores in Scorecard.
Claude posts a summary to the team's Microsoft 365 channel, highlighting any metric below threshold.
Failing results open a ticket in the ticketing system, assigned to the responsible engineer for review.
OutcomeRelease decisions rest on a consistent, recorded set of evaluation results. Teams typically spend less time compiling reports and catch quality drops earlier in the release cycle.
How we set it up
Free automation audit
We review how your teams currently run AI evaluations and identify the workflows most worth automating with Scorecard MCP. You leave with a clear list of priorities and an estimate of effort.
Secure connection
We connect Claude to Scorecard MCP using Dynamic Client Registration OAuth sign-in, so each user authorises access through their own account. Access is set to the least permissions each workflow needs.
Workflow design
We design the end-to-end workflow across Scorecard and your other systems such as Microsoft 365, CRM, ERP or ticketing. We agree which actions run automatically and which need a person's approval.
Testing with real past cases
We test the workflow against real past evaluation runs and testsets from your own projects. We check accuracy, refine prompts and confirm that approvals and audit logs work as intended.
Launch, monitoring and training
We launch the workflow with monitoring and audit logs in place, then train your team on how to use it and how to handle exceptions. We stay available for follow-up adjustments after launch.
Who benefits
- Machine learning engineers
- Spend less time on manual reporting and more time improving models, with evaluation results available through simple questions.
- AI product managers
- Get clear, current summaries of evaluation runs before release decisions, without waiting for others to compile them.
- Quality and compliance leads
- Maintain a consistent, auditable record of scores, annotations and approvals across every evaluation project.
- Operations and IT heads
- Connect evaluation work to existing ticketing and communication tools, with permissions they control and can review.
Security and control
- Access uses Dynamic Client Registration OAuth sign-in, so each person connects with their own account rather than shared credentials.
- Permissions are limited to the least access each workflow needs, and can be reviewed and revoked at any time.
- Actions that change records, such as starting runs or updating scores, can require a named person's approval before they take effect.
- Audit logs record what Claude did, when, and on whose instruction, supporting internal review.
- Human reviewers remain responsible for final scores and release decisions.
For teams in the US, UK, India and the Middle East
Organisations in the United States and the United Kingdom can align evaluation workflows with their existing data protection and governance rules. Indian organisations can design their workflows to meet the Digital Personal Data Protection Act, 2023. Organisations in the UAE and Saudi Arabia can meet local data protection requirements in the same way. We choose the hosting region for any custom application to meet your data residency and compliance needs.
Learn more about Claude and MCP
Anthropic's AI model and assistant.
Secure links between Claude and business tools.
How engineers build and change software with Claude.
Hosted agents that run multi-step work.
The open standard Claude uses to connect to tools.
Frequently asked questions
What does the Scorecard MCP server do?
The Scorecard MCP server lets AI tools such as Claude securely work with your Scorecard account. Typically it allows reading evaluation projects, testsets, metrics and runs, recording scores and annotations, and searching documentation. The exact actions available depend on the permissions you grant.
Is Scorecard MCP secure?
Security depends on how it is set up. Access uses Dynamic Client Registration OAuth sign-in, so each user authorises their own connection. Permissions can be limited to what each workflow needs, and approvals and audit logs can be added to sensitive actions.
What can Claude do with Scorecard once connected?
Claude can typically answer questions about your evaluation projects, summarise runs, help review and annotate outputs, record scores and search documentation. For example, it can produce a release summary comparing this week's runs with last week's.
How long does setup take?
Timing depends on how many workflows you want and how many systems they touch. A simple connection and single workflow can often be set up in a few weeks, including testing and training. We confirm a realistic timeline during the free automation audit.
What plan or permissions do I need?
You need an active Scorecard account with permission to access the projects you want Claude to work with. The specific plan requirements are set by Scorecard, so confirm your account's access during the audit. We will identify the least permissions each workflow needs.