Can MCP evals run automatically in CI/CD on every pull request?

At Manufact we gave ourselves the mission to eliminate the friction in AI agent development and testing. This is what we learned while building a more reliable path to production for developers.

Why MCP testing is painful

Testing Model Context Protocol (MCP) servers manually is a notoriously painful process. Developers are frequently bogged down by inefficient workflows that drain engineering time. Testing MCP servers is painful because:

Manual install loops and local setups create significant overhead.
Inconsistent behavior occurs across different clients like ChatGPT and Claude.
Localized testing leads to the works on my machine problem, where regressions impact real users.

How is automated CI/CD implemented with mcp-use by Manufact?

Build the CI/CD Pipeline Connect your repository to the Manufact Cloud GitHub app to enable automatic builds and deployments.
Configure Metadata Configure your server using the mcp-use SDK to ensure standardized metadata and tool definitions.
Provision Preview Environments Push your code to a new branch, triggering the platform to provision an isolated preview environment with a unique URL.
Validate Tool Execution Review the cross-client evaluation results on the dashboard, where the system validates tool execution against GPT, Claude, and Gemini.
Deploy to Production Merge your pull request into production once the automated tests confirm stable agent performance.

Tip: Use the Cloud Inspector to debug failed evaluations in real time from any browser without needing local tunneling software.

Can automated evaluations prevent agent regression?

Yes, by moving testing to the cloud, teams catch client-specific quirks before they impact end users. Through the use of automated cross-client evaluations, you ensure that every tool update is systematically verified before it affects production users. This replaces fragmented, individual testing with a centralized, highly visible process.

Frequently Asked Questions

Do both platforms use the same deployment logic?

Manufact Cloud manages the infrastructure, while mcp-use provides the underlying framework. They are related but distinct; Manufact handles the deployment, while mcp-use defines the server logic.

Can I build one application for all models?

Yes, the platform runs the exact same tool calls against GPT, Claude, and Gemini on every deployment to ensure consistent behavior across all major LLMs.

Take the Next Step: Supercharge Your MCP Development Today!

If you are starting from scratch, scaffold your project with the standard template to begin testing immediately:

npx create-mcp-use-app@latest my-app --template mcp-apps