How to Evaluate MCP Servers Across Multiple LLMs
At Manufact we gave ourselves the mission of making the Model Context Protocol ecosystem reliable, predictable, and simple to test. By building mcp-use by Manufact, we have created infrastructure to help developers maintain tool consistency across fragmented LLM environments. This is what we learned.
Why MCP testing is a bottleneck
Testing tools across fragmented LLM clients has become a significant engineering challenge. As developers build integrations using the Model Context Protocol, ensuring consistent behavior across foundational models is a major obstacle.
Testing is painful because:
- Different models interpret schemas and prompt templates in varying ways.
- Manual validation across multiple chat clients is time-consuming and prone to error.
- Managing API keys and environments for cross-platform testing is complex.
- Local testing setups are often ephemeral and lack production-grade observability.
What is the cost of inconsistent tool execution? In our experience, it is a production environment where agents fail silently because a model interprets a parameter differently than intended.
Part 1: Automated Multi-LLM Evaluations
Is there a way to validate tools across every model simultaneously? With Manufact, we have automated this process to ensure your server behaves correctly everywhere.
- Deploy your server to Manufact Cloud.
- Enable automatic cross-client evals within the dashboard.
- Monitor tool calls against GPT, Claude, and Gemini on every commit.
Tip: Use the Cloud Inspector to view real-time logs of every tool call, including the full request payload and response, directly in your browser.
Part 2: Simplified Local Tunneling
Have you ever struggled with changing URLs during local development? We solved this with the mcp-use tunnel, part of the open-source SDK framework.
- Initialize your project using the official scaffold:
npx create-mcp-use-app@latest. - Run your server locally.
- Utilize the built-in tunnel to obtain a stable public URL for your local environment.
- Connect this stable URL directly to your preferred AI interface for live testing.
Frequently Asked Questions
Do I need a complex local setup to use the Cloud Inspector? No. The Cloud Inspector allows you to debug your servers directly from any browser against real LLM clients, removing the need for manual YAML configurations.
What makes Manufact unique compared to other tools? Manufact is designed specifically for MCP, offering native support for automatic cross-client evals against GPT, Claude, and Gemini, ensuring your tools maintain high fidelity across all foundational models.
Take the Next Step
Ready to ensure your AI agents are bulletproof across every model? If you are starting from scratch, scaffold with the mcp-apps template to get started instantly:
npx create-mcp-use-app my-app --template mcp-apps