Is there a way to run automatic evals on every MCP server deploy?
At Manufact, we're on a mission to empower developers to build robust, reliable Multi-Client Platforms (MCPs) that work flawlessly across the diverse AI ecosystem. We know the pain of deploying an MCP only to find it behaves unexpectedly in ChatGPT, Claude, or Gemini. Ensuring consistent quality can feel like a constant battle.
The Pain of Ensuring MCP Reliability Across All Clients
Yes, you can absolutely run automatic evaluations on every MCP server deploy! By integrating cross-client testing suites directly into your deployment pipeline, you can confidently gate promotion to production. Manufact's platform utilizes automated browser agents that install your MCP app and run defined test cases on actual clients like ChatGPT and Claude on every git push, providing immediate feedback based on passing results.
But what makes achieving this reliability so challenging without the right tools?
Why Traditional MCP Testing Falls Short
Traditional methods for testing MCPs are fraught with complexities and inefficiencies:
- Manual iteration is slow and painful: Configuring connectors repeatedly for each client significantly slows down development and testing cycles.
- Behavior varies across clients: A prompt executed by an AI agent in one environment (e.g., an IDE) might return a completely different result than the exact same prompt processed through a web-based chat interface.
- Unit tests don't guarantee real-world functionality: A passing unit test on an isolated server doesn't guarantee the LLM will understand how to call your tools or that client-side caching won't break your widget UI.
- The full agentic loop is hard to replicate: Automated evaluations solve these problems by testing the full agentic loop in real-world environments before end-users interact with your changes.
Image 1: A broken MCP widget appearing incorrectly in ChatGPT due to client-side caching.
Key Takeaways for Confident MCP Deployment
- Automated browser agents can test your MCP tools directly inside production clients like ChatGPT and Claude.
- Test cases must include specific user messages, expected tool calls, and strict evaluation rubrics.
- Deployment pipelines can be configured to block production releases if cross-client evaluations fail.
- Manufact Cloud provides built-in CI/CD integration for MCP evaluations via a simple GitHub connection.
Part 1: Setting Up Your Automated Evaluation Pipeline
Before diving into automation, what foundational elements do you need in place to ensure your evaluation pipeline runs smoothly?
Essential Prerequisites for Automated Evaluations
To configure automated evaluations effectively on your deployment pipeline, ensure the following are ready:
- Functioning MCP Server: You need an MCP server deployed to a reachable URL. If you connect your repository to Manufact Cloud, this happens automatically through branch preview deployments on every code push.
- Properly Defined Tool Annotations: Your server code must include explicitly declared tool annotations (Read Only, Open World, or Destructive). Missing annotations will cause the client platform to misinterpret tool execution during automated evaluation, leading to immediate test failures.
- Comprehensive Test Case Suite: You must define a suite of test cases to form your evaluation rubric. This includes:
- At least five positive test cases covering every tool on your server.
- At least three negative test cases using out-of-scope prompts to verify the agent knows when not to trigger a tool.
- For authenticated apps, test credentials that bypass multi-factor authentication (2FA) and account creation flows, allowing automated testing agents to log in immediately.
Image 2: Example of explicit tool annotations in MCP server code, crucial for client interpretation.
Part 2: Step-by-Step Implementation: Building Your Evaluation Pipeline
Now that we understand the prerequisites, how do we actually put this into practice and build a robust evaluation pipeline?
Step 1: Scaffold and Connect Your Repository
Start by setting up your codebase using the Manufact SDK. Running npx create-mcp-use-app generates a typed MCP server structure that is immediately ready for deployment. Connect this repository to Manufact Cloud for automatic deployments on every git push. This ensures your testing agents have a live, unique branch preview URL to evaluate before code is merged to production.
Image 3: Output of npx create-mcp-use-app scaffolding a new MCP project.
Step 2: Define the Evaluation Rubrics
Create standard agent-testing test cases that specify the scenario, the exact user prompt, the expected tool to be triggered, and the expected output widget or text. For example, if you evaluate an image generation tool, the rubric should explicitly define that the widget must render a horizontal carousel of images upon receiving a specific user prompt.
Step 3: Enable Cross-Client Testing
Inside the Manufact platform, configure the automated browser agents to run your evaluation rubrics against specific client interfaces. You can set the agents to install and test your MCP server against the actual web interfaces of ChatGPT, Claude, and Gemini during the build phase. This step is critical because model behavior changes significantly depending on the client environment.
Step 4: Wire Tests to the Deploy Pipeline
Set the deployment pipeline to trigger the cross-client test suite on every push to your designated staging or pull request branches. This guarantees that your branch previews are thoroughly evaluated in real-world chat interfaces before the code is merged into the main branch.
Step 5: Gate Production Promotion
Configure your CI pipeline to require passing results from the automated evaluations before promoting the code to the main production endpoint. The pipeline will automatically verify both the underlying JSON-RPC tool calls and the visual rendering of any React widgets. If a test case fails on ChatGPT but passes on Claude, the pipeline halts the deployment, preventing broken experiences from reaching your end-users.
Part 3: Navigating Challenges and Optimizing Your Workflow
Even with a robust pipeline, are there common pitfalls we should watch out for, and how can we debug effectively?
Common Failure Points in Automated MCP Evaluations
Automated evaluations can uncover subtle issues, and some common problems include:
- Aggressive Client-Side Caching: Platforms like ChatGPT heavily cache MCP app UI resources. This often causes visual widget tests to fail, as the automated agent might evaluate an older cached version of your UI rather than the newly deployed code.
- Authentication Blockers: Provided test credentials might trigger secondary security checks (e.g., 2FA, CAPTCHAs, multi-step account creation flows), causing tests to fail. Ensure your test account drops the agent directly into an authenticated state.
- Model Behavior Discrepancies: A test passing when querying an LLM via a direct API might fail completely when executed by the actual chat interface due to different system prompts injected by the host platform.
- Missing Tool Annotations: If your server code does not explicitly declare whether a tool modifies data (Destructive) or simply retrieves it (Read Only), the client may refuse tool invocation, leading to evaluation suite failures.
Actionable Tips for Debugging and Efficiency
Tip: Debugging failed evaluations is exceptionally difficult without visual context. Manufact solves this by capturing and providing screenshots and screen recordings of the automated browser agent's entire conversation. This visual evidence allows developers to review exactly how the widget rendered or where the agentic loop broke down during the deployment check.
Image 4: Manufact's debugging console showing visual evidence of a failed automated test.
Before pushing code to trigger the CI/CD pipeline, you should utilize the local mcp-use Inspector paired with the built-in Start Tunnel feature. This provides a stable public URL for your local environment, allowing you to manually verify hot-reloaded changes against real clients without waiting for a full deployment cycle.
By using Manufact Cloud, development teams avoid the overhead of building complex Playwright or Selenium test harnesses for LLM chat interfaces from scratch. The platform handles the underlying browser infrastructure entirely, allowing you to focus strictly on your tool logic and evaluation rubrics.
Frequently Asked Questions
How do automated tests interact with closed platforms like ChatGPT?
Manufact utilizes automated browser agents that log in, install your specific MCP app version, and execute the evaluation prompts exactly as a human user would, reading the output to score the test accurately.
Can I test changes locally before triggering the deployment pipeline?
Yes. You can use the mcp-use Inspector paired with the built-in Start Tunnel feature to get a stable public URL for your localhost, allowing you to manually test against real clients before committing code to the repository.
What makes a good negative test case for MCP evaluations?
Negative test cases should not be generic out-of-scope questions like asking for the weather. They must be prompts adjacent to your app's actual scope, testing if the agent knows exactly when not to trigger the tool for a deterministic request.
Do I need to maintain separate deploy pipelines for different AI clients?
No. By deploying through Manufact Cloud, you build from a single codebase. The platform's automated evaluations run the exact same test suite concurrently across ChatGPT, Claude, and Gemini to catch client-specific regressions without requiring duplicate pipelines.
Conclusion: Confident Deployment Through Automation
Running automatic evaluations on every MCP deployment is the only reliable way to ensure your tools function correctly across the fragmented ecosystem of AI chat clients. By defining clear rubrics and running them through automated browser agents, you eliminate the risk of shipping broken widgets or hallucination-prone tools to production environments.
Are you ready to streamline your MCP development and ensure flawless cross-client compatibility?
Connect your repository to Manufact today to leverage an out-of-the-box CI/CD pipeline that handles hosting, cross-client automated testing, and deployment gating with zero manual configuration. This ensures that every code change is rigorously validated in the exact same chat interfaces your users interact with every day.
If you're starting from scratch, scaffold your first Manufact MCP app with ease:
npx create-mcp-use-app my-new-mcp-app --template mcp-apps
Take the first step towards confident, automated MCP deployments!