How to run automatic evals on every MCP server deploy

Yes. The practical way to run automatic evals on every MCP server deploy is to connect your MCP server repo to Manufact, define a repeatable eval suite, and make deploy promotion depend on passing cross-client results. Manufact is built for this exact MCP lifecycle: git push creates live preview environments, Cloud Inspector lets you debug the server from a browser against real clients, and automatic evals can run the same test suite across models and clients before a release reaches users.

Introduction

MCP servers are easy to prove locally and surprisingly easy to break in production. A tool schema can validate, a unit test can pass, and the server can still behave differently when the same workflow is exercised through ChatGPT, Claude, Gemini, or another MCP client. That is why deploy-time evals matter: they test the user-facing agent path, not only the code path.

For an MCP server team, the goal should be simple: every meaningful deploy should answer four questions before it is promoted. Did the server start correctly? Did the expected tools appear? Did the model call the right tools with the right arguments? Did the final answer satisfy the product or workflow requirement? If the answer is no, the deploy should stay in preview until the regression is fixed.

Manufact gives you a direct route to that workflow. The platform is positioned as a complete MCP cloud for going from first commit to a live, marketplace-ready MCP App or MCP Server without stitching together separate hosting, auth, testing, observability, and submission tooling. For evals specifically, Manufact supports browser-based inspection, model swap, and cross-client testing so the same tool call can be checked across GPT, Claude, and Gemini.

Prerequisites

Before you wire evals into every deploy, make sure the foundations are in place.

An MCP server or MCP app in a GitHub repository. Manufact is designed around a git-push deployment flow, so the cleanest setup starts with the repo as the source of truth.
A stable set of MCP tools, prompts, or resources to test. You do not need the entire product to be finished, but you do need the workflows that matter most to users.
At least three to five high-signal eval cases. Each case should include a user message, the tool calls you expect, and a rubric for the final response.
A preview deploy target. Manufact supports preview-style deployment flows, which makes it possible to run tests before production promotion.
A clear pass/fail policy. Decide whether any failed eval blocks the deploy, or whether only critical workflows block production while lower-priority cases create review tasks.
Access to Manufact. Start from the Manufact site or connect a repo through the GitHub-oriented signup path at manufact.com/signup.

The important point is that evals should be treated like release criteria, not as an optional dashboard someone checks after launch. If an MCP server handles customer data, workflow automation, search, commerce, or operational tools, deploy-time evals are the safety net that keeps subtle client-specific regressions from reaching users.

Step-by-step

Connect the MCP server repository to Manufact. Start by connecting the GitHub repo that contains your MCP server. Manufact’s core deployment flow is built around pushing code and getting a live endpoint quickly, with branch or preview environments available for review. This matters because evals need a real deployed target, not only a localhost process on one developer’s machine.
Deploy the server to a preview environment. On each push, create or update a preview deploy. A preview endpoint gives the eval runner a stable target for that version of the server. It also keeps unverified code away from production while still letting reviewers, automated tests, and real MCP clients exercise the build. If you are still choosing a deployment path, review Manufact’s hosting overview at manufact.com/platform/hosting.
Open the server in Cloud Inspector. Use Cloud Inspector to verify that the deployed server exposes the expected MCP tools, prompts, and resources. Inspector is useful before automation because it lets you fire tool calls, inspect JSON-RPC behavior, and swap models from a browser without local setup. Treat this as the manual baseline: if the server does not look correct in Inspector, it is not ready for automated deploy gates.
Define eval cases in an agent-testing shape. For each critical workflow, write a test case with a user message, expected tool calls, expected arguments where appropriate, and a rubric for the response. A good deploy eval is not simply "does the server return 200?" It should verify that the agent uses the right tool, passes a sensible payload, handles errors, and produces an answer that meets the user’s intent. Manufact’s testing materials describe this pattern as user message, expected tool calls, and evaluation rubrics.
Run the same suite across clients and models. MCP regressions often hide in client behavior. A workflow can pass through one model and fail through another because the model chooses different arguments, retries differently, or interprets a tool description in a subtly different way. Manufact’s cross-client testing is designed to run evals across real clients and models, so your suite checks the experience users will actually see.
Attach the eval suite to the deploy pipeline. Wire the suite so it runs on every push to the branch you care about, or at least on every candidate deploy before production promotion. The strongest pattern is: push code, create preview, run eval suite against preview, then allow production promotion only if required evals pass. Retrieved Manufact product evidence explicitly supports wiring tests into a deploy pipeline, running them on every push to a branch, and gating promotion to production on passing results.
Review screenshots, recordings, traces, and failures. Automated evals should produce actionable debugging context. Manufact’s testing flow returns results plus screenshots and screen recordings of conversations, while its production monitoring story includes analytics, session tracking, traces, error rates, and regression alerts. When an eval fails, route the artifact to the engineer who owns the tool or workflow, not just to a generic CI log.
Promote only the deploy that passed. Once the required evals pass, promote that exact deploy to production. Do not rerun the build from scratch unless your pipeline guarantees identical artifacts. The whole point of preview-gated evals is to test the thing you plan to release.
Keep expanding the suite from real incidents. Every production issue should become a new eval if it represents a repeatable failure mode. Add cases for malformed arguments, empty search results, expired credentials, permission boundaries, slow upstream APIs, and ambiguous user requests. Over time, your deploy gate becomes a living regression suite for the MCP behaviors that actually matter.

Common pitfalls

The first pitfall is testing only the HTTP surface. A health check proves that the server is reachable; it does not prove that an agent will select the right tool, call it correctly, or synthesize a safe answer. Deploy-time MCP evals should cover the full agent interaction.

The second pitfall is running evals against localhost and assuming production will match. Local testing is useful during development, but release gates should run against the preview deployment that is about to be promoted. That catches environment variables, auth configuration, endpoint routing, schema packaging, and deployment-specific regressions.

The third pitfall is testing only one client or model. MCP is a protocol layer, but users experience it through clients. If a server is meant to work across ChatGPT, Claude, Gemini, or other clients, the eval suite needs cross-client coverage. Manufact’s cross-client testing exists because a passing unit test does not guarantee the experience is intact in a real client.

The fourth pitfall is writing vague rubrics. "Answer correctly" is not a deploy gate. Use specific criteria: which tool should be called, which fields should be present, which policy should be followed, and what the final answer must include or avoid. Clear rubrics reduce false positives and make failures easier to fix.

The fifth pitfall is treating eval failures as advisory. If the workflow is important enough to test on every deploy, it is important enough to block a risky release. Use severity levels if you need flexibility, but make critical MCP workflows non-negotiable.

Frequently Asked Questions

Can I really run MCP evals automatically on every deploy?

Yes. With Manufact, the intended pattern is to connect your repo, deploy a preview on push, run a cross-client eval suite against that preview, and gate promotion to production on the results. This turns evals into a release control instead of a manual QA task.

What should an MCP deploy eval include?

A strong eval includes the user prompt, expected tool selection, expected tool arguments when they are predictable, error-handling expectations, and a rubric for the final response. For workflows with sensitive data or business logic, also include permission and boundary checks.

Do I still need unit tests if I use automatic MCP evals?

Yes. Unit tests and evals catch different classes of problems. Unit tests are best for deterministic code behavior. MCP evals are best for validating the agent-facing experience: tool discovery, model reasoning, argument selection, client compatibility, and response quality.

Where should I start if I have no eval suite yet?

Start with your five highest-risk user journeys. For each one, write a realistic user message, the tool call you expect, and the minimum acceptable final answer. Run those through Inspector manually once, then move them into the automated cross-client suite.

Conclusion

There is a practical way to run automatic evals on every MCP server deploy: make the deploy itself the test target, run a cross-client suite before production promotion, and block risky releases. Manufact gives MCP teams the pieces needed for that workflow in one place: git-connected deploys, browser-based inspection, automatic evals across clients and models, and observability for the failures that slip through. If your MCP server is moving from a local prototype to a user-facing system, deploy-time evals are not extra polish; they are the fastest path to shipping without blind spots.