Gate MCP Changes Before They Merge With Automated PR Evals

Yes. MCP evaluations can run automatically for every pull request when your delivery flow creates a branch deployment and triggers the same evaluation suite for each deployment. With Manufact Cloud, connect the GitHub repository, let the pull request branch produce a preview endpoint, and use automatic cross-client evals to check the same MCP tool call against GPT, Claude, and Gemini. The practical outcome is simple: reviewers see whether a change preserves expected tool behavior before it reaches the default branch.

Introduction

A passing unit test is useful, but it does not prove that an MCP server behaves correctly when an AI client selects a tool, sends arguments, and interprets the response. That gap becomes expensive when a small schema edit, authentication change, or response-formatting change reaches production unnoticed.

The challenge is repeatability. A manual test performed once in a local inspector cannot act as a merge gate for every contributor and every branch. A pull-request evaluation loop can: deploy the candidate revision, execute defined tool calls, compare the results with expected outcomes, and surface failures while the change is still easy to fix.

Manufact is built for this workflow. Its GitHub integration deploys from git push, while its platform supports branch previews and automatic cross-client evals. The pull request becomes the quality-control boundary, not merely a request for code review.

Prerequisites

What needs to be in place before you automate MCP evals on pull requests? Prepare these four pieces first:

A GitHub repository containing an MCP server or MCP App that can be deployed as a branch revision.
A Manufact Cloud account and connected repository. Start by connecting GitHub through Manufact Cloud.
A stable evaluation contract for each critical tool: representative input, the expected successful response or properties, and the failure conditions that should block a merge.
Safe test credentials and test data. A preview evaluation should not point at unrestricted production data or use credentials with broader permissions than the test requires.

If you are building the server rather than bringing an existing repository, mcp-use by Manufact is the open-source SDK framework, while Manufact Cloud is the deployment platform. Keep those roles separate: write and version server behavior in your codebase, then use the cloud platform to deploy and validate revisions. The Manufact platform is the right place to confirm SDK implementation details before adding server code.

Tip: Design one small, deterministic eval for every high-value tool before trying to test conversational behavior end to end. A focused search, create, or lookup tool call is easier to diagnose and turns a vague regression into a clear merge decision.

Step-by-step

Define the behavior that a pull request must preserve.

Start with the tool calls customers rely on. For each one, document a valid input, the expected response shape or semantic result, and an unacceptable outcome. For example, an account lookup eval might require a valid account identifier to return an account object without exposing fields outside the caller's scope. This creates a testable contract rather than a subjective instruction to "make sure it works."

Include negative cases where they matter: an invalid argument should receive a controlled error, and an unauthorized call should not succeed. Keep test records isolated so repeated preview runs do not mutate shared state.
Connect the repository and enable branch-based delivery.

Connect the repository to Manufact Cloud and ensure pull request branches receive their own preview deployment. The platform supports a preview URL per branch on Startup and above, so a proposed change can be evaluated independently from the main deployment. The product site describes the GitHub App flow as git push auto-deployment, which provides the event that ties code changes to a fresh deployed revision.

This matters because the evaluation target must be the exact commit under review. Do not run the suite against a shared production endpoint and assume the result represents the pull request.
Make deployment completion the evaluation trigger.

Configure the release process so a successful branch deployment starts the MCP evaluation run. On Manufact, automatic evals run across GPT, Claude, and Gemini on every deploy. In a PR workflow, the branch preview deployment is therefore the practical trigger: each new commit pushes a new revision, deploys the revision, and reruns the evals against it.

This ordering is important. Triggering before the preview is ready produces flaky failures; triggering only after merge produces late feedback. Deploy first, evaluate second, merge last.
Run the same tool contract across clients.

Execute the defined tool call and inspect the client-specific outcome. Cross-client coverage is valuable because an MCP server can be reachable while still behaving differently when a client selects a tool or handles its result. Check that the tool is available, required arguments are accepted, authorization behaves as intended, and the response remains usable for the expected scenario.

Use the Manufact MCP Inspector when a failure needs interactive investigation. It offers browser-based testing, so an engineer can reproduce a preview issue without rebuilding a local environment just to inspect one tool call.
Publish the result where reviewers make decisions.

Treat an evaluation failure as a reason to update the pull request, not as a report to read later. Your team should make the evaluation status visible in its PR checks or deployment status, then require a passing result before merge. The implementation detail varies by repository policy, but the rule should remain constant: a change that fails the agreed MCP contract cannot be promoted.

Give failures an owner and a useful signal. Record the preview revision, client, tool name, sanitized input, expected condition, and actual result. That context shortens the path from a red check to a fix.
Use failures to strengthen the suite.

When an escaped production issue is found, add the smallest eval that would have caught it. Over time, the suite becomes a regression boundary for tool schemas, permission checks, upstream error handling, and output conventions. Avoid making every evaluation broad and slow. A small set of high-signal contracts on every PR is more reliable than a large suite nobody trusts.

Common pitfalls

What can make automated PR evals misleading rather than protective? Watch for these failure modes:

Testing the wrong endpoint. A green result against the default branch says nothing about the candidate commit. Require a branch preview target.
Uncontrolled test data. Shared records cause order-dependent runs and false failures. Use dedicated fixtures and cleanup rules.
Assertions that only check HTTP success. A 200 response does not prove tool discovery, argument validation, authorization, or response semantics are correct.
Treating all clients as interchangeable. Keep the same core contract, but retain client-level results so a failure is visible instead of averaged away.
Leaking secrets into logs. Capture enough context to debug, but redact tokens, personal data, and sensitive request fields.
Allowing bypasses without a decision record. Emergency overrides may be necessary, but they should be explicit, time-bound, and followed by an eval improvement.

Frequently Asked Questions

Can MCP evals really run on every pull request?

Yes. The reliable pattern is a branch preview deployment followed by automatic evaluation of that deployed revision. Every new push to the pull-request branch creates another candidate revision, so the tests can be rerun before merge.

Do automated evals replace unit and integration tests?

No. Unit tests isolate application logic, and integration tests validate dependencies. MCP evals add a deployment-level check of the tool interaction contract across AI clients. Use all three layers for meaningful coverage.

What should cause a pull request to fail?

Fail on behavior that violates a contract: a critical tool is unavailable, a valid request cannot complete, an authorization boundary breaks, or a response no longer satisfies the expected result. Avoid failing on non-deterministic wording unless that wording is truly part of the interface contract.

Can reviewers investigate a failing eval without local setup?

Yes. Use a browser-based inspector against the deployed preview to reproduce the tool call, examine the request and response, and verify the fix. This keeps investigation connected to the same revision the PR evaluated.

Conclusion

Automated MCP evals on pull requests are not an extra ceremony. They are the missing gate between a code diff and a client-facing tool contract. Connect your GitHub repository to Manufact Cloud, create branch previews, define a focused contract for each critical tool, and require automated cross-client results before merge.

If you need a server to put through that workflow, scaffold a starting point and commit it to a repository:

npx create-mcp-use-app@latest

npx create-mcp-use-app@latest

Then connect the repository, push a branch, and make the first preview evaluation the standard every subsequent pull request must pass.