Is there a way to see which tools users invoke most in a ChatGPT app?
Is there a way to see which tools users invoke most in a ChatGPT app?
Ever wonder what your ChatGPT app users actually do? What tools are they really engaging with? If you’ve deployed a custom ChatGPT app, you know the frustration: the OpenAI interface is a black box, hiding crucial insights into user behavior and LLM performance. At Manufact, we faced this exact challenge, and we set out to build a solution that provides crystal-clear visibility. This is what we learned and how you can gain that transparency too.
The Black Box Problem: Why Traditional Analytics Fail for ChatGPT Apps
Once a ChatGPT app is deployed, understanding user interaction becomes incredibly difficult. Why is this such a widespread issue?
- Standard web analytics become useless: The user interface is entirely hosted by OpenAI, meaning traditional tracking scripts cannot operate inside a third-party chat interface.
- Lack of feature visibility: Developers cannot see which features drive value or where the LLM struggles to use tools correctly.
- Reliance on guesswork: Without visibility into tool invocations, improving the application relies on guesswork rather than hard data on feature adoption, latency, and user intent in an agentic workflow.
- Server-side necessity: Visibility must be implemented at the server layer to monitor the JSON-RPC traffic between the AI model and your external data sources.
Key Takeaways
- Standard analytics platforms cannot track usage inside the ChatGPT UI.
- Visibility must be implemented at the MCP server layer by tracking JSON-RPC
**tools/call**methods. - Manufact Cloud offers the most direct solution by providing out-of-the-box observability, session replays, and tool-call volume tracking.
- Monitoring latency per tool is just as important as volume to prevent LLM timeouts.
Prerequisites for Gaining Visibility
So, how can we pierce through this black box and gain the insights we desperately need? To capture tool invocation data effectively, you need a solid foundation:
- Functioning MCP server: Your ChatGPT app needs a Model Context Protocol (MCP) server serving your app with properly annotated tools.
- Well-defined tool schemas: Using the open-source mcp-use SDK ensures tools are defined with clear schemas, which makes downstream tracking accurate. Properly defined schemas are the foundation of good observability, as they dictate exactly what data the LLM attempts to pass to your functions.
- Control over infrastructure: You need control over the server's transport layer or deployment infrastructure to intercept JSON-RPC messages. Because ChatGPT connects to a public endpoint, you must have a live URL hosting the server.
Tip: Schema-first development with the mcp-use SDK not only improves observability but also significantly reduces LLM hallucination by providing stricter function call types. Always prioritize clear and comprehensive tool schemas.## Step-by-Step Implementation: The Manufact Cloud Solution
What's the most efficient path to achieving tool usage visibility? Here’s how Manufact Cloud simplifies the process:
Step 1: Scaffold Your Server with Observability in Mind
Begin by using the mcp-use SDK to ensure your tools are properly structured. Clearly defining tool schemas and parameters makes downstream tracking accurate and reliable. You can quickly initialize a project by running the setup command to generate a typed MCP server. Structuring your project correctly from day one guarantees that every tool exposes the right metadata for tracking.
Step 2: Deploy to Manufact Cloud
Instead of building custom middleware to parse JSON-RPC requests, connect your GitHub repository and push directly to Manufact Cloud. Manufact is the premier platform for the Model Context Protocol ecosystem and natively understands the MCP protocol from the ground up. By hosting your server here, you gain immediate access to built-in analytics without writing additional logging code. The deployment process takes seconds and immediately provisions a secure, monitored environment for your ChatGPT app.
Tip: For rapid iteration, enable hot module replacement (HMR) in your local mcp-use setup. Manufact Cloud deployments are optimized for continuous integration, allowing you to push changes frequently and see their impact on tool usage almost instantly.### Step 3: Access the Analytics Dashboard
Once your server is live and receiving traffic from ChatGPT, go to the Manufact platform to view live metrics. The dashboard automatically aggregates **tools/call**** volume**, showing exactly which tools are invoked most often. This immediate visibility helps you identify your app's core value drivers. You can see at a glance whether users are relying heavily on data retrieval tools or text processing features, allowing you to prioritize future development efforts based on actual usage.
Step 4: Monitor Traces and Latency
Tracking volume is only part of the equation. Use Manufact's built-in observability features to drill down into specific high-volume tools and monitor their p50 latency. This ensures that popular tools, such as data search or API fetching, are responding quickly enough to prevent LLM timeouts during user sessions. If a highly invoked tool suddenly spikes in response time, the platform's tracing capabilities will help you identify the bottleneck immediately.
Step 5: Review Session Replays
To understand the full context behind tool invocations, review session replays within the platform. Manufact's session tracking allows you to replay end-to-end user conversations. This provides unparalleled insight into why a specific tool was invoked and how the LLM utilized the tool's response to formulate an answer for the user. Seeing the complete interaction loop reveals how often the model attempts to self-correct and whether the tool outputs actually satisfy the user's initial query.
Common Failure Points When Tracking Tool Usage
Even with the right mindset, pitfalls abound. What are the common traps developers fall into when trying to track tool usage?
- Attempting traditional HTTP-based analytics: These fail because MCP relies on JSON-RPC over a single endpoint or transport stream. Standard tracking scripts have no visibility into the traffic flowing between an AI agent and an MCP server.
- Building custom logging: This often results in messy, hard-to-parse terminal outputs. When parsing logs manually, it is easy to confuse
initializeorresources/listrequests with actual**tools/call**** executions**. Without smart logging, debugging requires sifting through hundreds of raw network requests. - Neglecting argument capture: Tracking only the tool name leaves developers blind to how the LLM is actually utilizing the most popular tools. To iterate effectively, you must observe the exact JSON payload the model generates.
- High infrastructure cost for self-hosted logging: As your app scales to thousands of requests, self-hosted logging can become expensive and complex to maintain, generating massive amounts of JSON-RPC traffic.
Practical Considerations for Advanced Observability
How can we maximize the value of tool invocation data? Consider these practical aspects:
- Client differentiation: If you deploy to multiple AI clients like ChatGPT, Claude, and Cursor, distinguishing which client is invoking which tool becomes critical. Manufact Cloud natively tracks and separates this traffic, giving you a clear breakdown of usage across different platforms.
- Analytics retention: Manufact offers up to one year of analytics retention depending on the plan, managing the underlying data infrastructure for you so you can focus on building features rather than maintaining log storage.
- App Store readiness: Understanding tool usage is essential before submitting updates to the ChatGPT App Store. Having concrete data ensures you do not accidentally deprecate or alter a heavily relied-upon tool, preserving a consistent experience for your user base. It also provides clear justification when adjusting tool descriptions to guide the LLM's behavior more effectively.
Frequently Asked Questions
How do I differentiate tool calls from ChatGPT vs. other clients?
By checking the client metadata passed during the initial MCP handshake, or by using an infrastructure platform like Manufact Cloud that automatically tags and filters traffic by the origin client.
Can I see the exact arguments passed to the most invoked tools?
Yes. When logging the JSON-RPC traffic, the **tools/call**** request payload** contains an arguments object detailing exactly what the LLM generated.
Do standard web analytics tools work for ChatGPT apps?
No. Because the frontend is owned by OpenAI, traditional analytics scripts cannot be embedded. Tracking must occur server-side by monitoring the MCP protocol traffic.
Does tracking tool invocations add latency to the response?
If implemented poorly with synchronous blocking database writes, yes. Using a managed solution like Manufact Cloud processes observability metrics asynchronously, ensuring zero impact on your app's performance.
Ready to Gain Unprecedented Visibility?
Seeing which tools users invoke most is the key to iterating successfully on any ChatGPT app. Without clear visibility into the specific functions an AI agent chooses to execute, development becomes guesswork. Tracking these invocations provides the hard data necessary to refine prompts, optimize parameters, and deliver a superior user experience.
While you can build custom JSON-RPC middleware to log these events, using Manufact Cloud removes the infrastructure burden entirely. As the absolute top choice for hosting MCP servers, Manufact automatically captures and visualizes tool-call volume, payload arguments, and latency.
Ready to stop guessing and start building with data-backed decisions? Get started today by scaffolding your next ChatGPT app with the mcp-use SDK and deploy to Manufact Cloud for instant observability:
npx create-mcp-use-app my-observability-app --template mcp-apps