Playground AI

Playground AI AI

Playground AI AI

Playground AI AI

Playground AI AI

Playground AI AI

Playground AI AI

Playground AI AI

Playground AI AI

Playground AI AI

Playground AI AI

I built a desktop app to inspect, debug, and reuse the MCP tools you make with Claude

Hi everyone, If you use Claude Code or Claude Desktop with MCP tools, you’ve probably run into this problem. Claude is incredible at generating tool logic quickly. But as soon as the tool is created: Did it actually execute correctly, or is the AI hallucinating? What arguments did Claude actually pass to it? If it failed, why? How do I reuse this tool outside of this specific chat session? Debugging MCP tools just by retrying prompts in the chat interface is incredibly frustrating. To solve this, I built Spring AI Playground — a self-hosted desktop app that acts as a local Tool Lab for your MCP tools. What it does: Build with JS: Take the tool logic Claude just wrote, paste it in, and it works immediately. Built-in MCP Server: It instantly exposes your validated tools back to Claude Desktop or Claude Code. Deep Inspection: See the exact execution logs, inputs, and outputs for every single tool call Claude makes. Secure: Built-in secret management so you don't have to paste your API keys into Claude's chat. The goal is to give the tools Claude generates a proper place to be validated and reused, instead of staying as one-off experiments. It runs locally on Windows, macOS, and Linux (no Docker required). Repo: https://github.com/spring-ai-community/spring-ai-playground Docs: https://spring-ai-community.github.io/spring-ai-playground/ I'd love to hear how you are all currently handling tool reuse and debugging when working with Claude. submitted by /u/kr-jmlab [link] [comments]

I made my agent 34.2% more accurate by letting it self-improve. Here’s how.

Edit: I rewrote everything by hand! Everyone I know collects a lot of traces but struggles with seeing what is going wrong with the agent. Even if you setup some manual signals, you are then stuck in a manual workflow of reading the traces, tweaking your prompts, hoping it’s making the agent better and then repeating the process again. I spent a long time figuring out how to make this better and found the problem is composed of the following building blocks with each having its technical and design complexity. Analyzing the traces. A lot can go wrong when trying to analyze what the failures are. Is it a one off failure or systematic? How often does it happen? When does it happen? What caused the failure? Currently this analysis step is missing almost entirely in observability platforms I’ve worked with and developers are resorting to the process I explained earlier. This becomes virtually impossible with thousands to millions of traces, and many deviations cause by the probabilistic nature of LLMs never get found because of it. The quality of the analysis can be/is a bottleneck for everything that comes later. Evals. Signals are nice but not enough. They often fail and provide a limited understanding into the system with pre-biasing the system, since they’re often set up manually or come generic out of the box. Evals need to be made dynamically based on the specific findings from step one in my opinion. They should be designed as code to run on full databases of spans. If this is not possible however, they should be designed through LLM as a judge. Regardless the system should have the ability to make custom evals that fit the specific issues found. Baselines. When designing custom evals, computing baselines against the full sample reveal the full extent of the failure mode and also the gaps in the design of the underlying eval. This allows you to reiterate on the eval and recategorize the failures found based on importance. Optimizing against a useless eval is as bad as modifying the agent’s behavior against a single non-recurring failure. Fix implementation. This step is entirely manual at the moment. Devs go and change stuff in the codebase or add the new prompts after experimenting with a “prompt playground” which is very shallow and doesn’t connect with the rest of the stack. The key decision in this step is whether something should indeed be a prompt change or if the harness around an agent is limiting it in some way for example not passing the right context, tool descriptions not sufficient etc. Doing all this manually, is not only resource heavy but also you just miss all the details. Verification. After the fixes, evals run again, compute improvements and changes are kept, reverted or reworked. Then this process can repeat itself. I automated this entire loop. With one command I invoke an agentic system that optimizes the agent and does everything described above autonomously. The solution is trace analyzing through a REPL environment with agents tuned for exactly this use case, providing the analysis to Claude Code through CLI to handle the rest with a set of skills. Since Claude can live inside your codebase it validates the analysis and decides on the best course of action in the fix stage (prompt/code). I benchmarked on Tau-2 Bench using only one iteration. First pass gave me 34.2% accuracy gain without touching anything myself. On the image you can see the custom made evals and how the improvement turned out. Some worked very well, others less and some didn’t. But that’s totally fine, the idea is to let it loop and run again with new traces, new evidence, new problems found. Each cycle compounds. Human-in-the-loop is there if you want to approve fixes before step 4. In my testing I just let it do its thing for demonstration purposes. Image shows the full results on the benchmark and the custom made evals. The whole thing is open sourced here: https://github.com/kayba-ai/agentic-context-engine I’d be curious to know how others here are handling the improvement of their agents. Also, how do you utilize your traces or is it just a pile of valuable data you never use? submitted by /u/Lucky_Historian742 [link] [comments]

[Project] JudgeGPT — open-source LLM-as-judge benchmarking tool with configurable scoring rubrics, CoT reasoning, and real-time GPU telemetry

Sharing a tool I built that lets you run your own LLM-as-judge evaluations locally, against any models you have running via Ollama. The core problem with LLM-as-judge that I tried to address: LLM judges are notoriously unreliable out of the box — position bias, verbosity bias, self-family bias (~5-7% score inflation when the judge shares a model family with the evaluated model), and leniency clustering in smaller models. Most local benchmarking tools just wrap a judge prompt around a response and call it a score. I wanted something more principled. What JudgeGPT does differently: 1. Scoring rubric with behavioral anchors Each of the 5 criteria (Accuracy, Clarity, Depth, Concision, Examples) has explicit behavioral descriptors at every score level — not just "1=bad, 5=good." This significantly reduces leniency clustering in sub-10B judge models. 2. Configurable judge model + system prompt from the UI You're not locked into one judge. Default is qwen2.5:7b (strong human correlation on judging benchmarks), but you can swap in any Ollama model and edit the system prompt at runtime without touching config files. This matters if you want to study judge-vs-judge disagreement. 3. Chain-of-thought before scoring The judge reasons freely first, then produces structured JSON scores informed by that reasoning. Forcing scores directly — without a reasoning pass — produces worse human alignment. The reasoning snippet is surfaced in the UI so you can audit it. 4. Human score blending You can add your own 5-star rating per response. It blends into the quality component of the combined score, so you're not entirely delegating evaluation to the judge. 5. Self-family bias warning When the judge model and evaluated model share a family, the UI flags it. It doesn't block you — sometimes you want to run it anyway — but it's there. Combined leaderboard score: TPS × 35% + TTFT × 15% + Quality × 50% Quality = average of judge score + human score (if provided). The weighting is configurable in the judge settings panel. Other features: 7 tabs: Run · Metrics · Responses · Overall · Stream Live · Playground · History Concurrent or sequential model execution (sequential = VRAM-saver mode) Real-time GPU telemetry (temp, power draw, VRAM) — Metal / ROCm / CUDA auto-detected — live sparklines during benchmark + summary in results Persistent benchmark history (SQLite) with one-click restore Download Manager for pulling models pre-benchmark Playground tab: side-by-side comparison of any two OpenAI-compatible endpoints (useful for comparing local vs API-hosted versions of the same model) Prometheus /metrics endpoint, PDF/JSON/CSV export Stack: FastAPI + Docker SDK (Python), React 18 + Vite, Recharts, Ollama, nginx. Runs via ./start.sh up. Repo: https://github.com/MegaBytesllc/judgegpt Genuinely curious if anyone has thoughts on the rubric design or better approaches to calibrating small-model judges. The behavioral anchors help but there's still meaningful variance in the 3B–7B range. submitted by /u/1T_Geek [link] [comments]

CodeGraphContext (An MCP server that indexes local code into a graph database) now has a website playground for experiments

Hey everyone! I have been developing CodeGraphContext, an open-source MCP server transforming code into a symbol-level code graph, as opposed to text-based code analysis. This means that AI agents won’t be sending entire code blocks to the model, but can retrieve context via: function calls, imported modules, class inheritance, file dependencies etc. This allows AI agents (and humans!) to better grasp how code is internally connected. What it does CodeGraphContext analyzes a code repository, generating a code graph of: files, functions, classes, modules and their relationships, etc. AI agents can then query this graph to retrieve only the relevant context, reducing hallucinations. Playground Demo on website I've also added a playground demo that lets you play with small repos directly. You can load a project from: a local code folder, a GitHub repo, a GitLab repo Everything runs on the local client browser. For larger repos, it’s recommended to get the full version from pip or Docker. Additionally, the playground lets you visually explore code links and relationships. I’m also adding support for architecture diagrams and chatting with the codebase. Status so far- ⭐ ~1.5k GitHub stars 🍴 350+ forks 📦 100k+ downloads combined If you’re building AI dev tooling, MCP servers, or code intelligence systems, I’d love your feedback. Repo: https://github.com/CodeGraphContext/CodeGraphContext submitted by /u/Desperate-Ad-9679 [link] [comments]

Claude Aria + Sonnet 5 Incoming?

Just today, on Anthropic's "CometAPI" page, It shows a real page with $2.4/Mtok pricing and a description. This is really exciting for I've been waiting long. submitted by /u/Affectionate-Peak156 [link] [comments]

So Claude is #1 in the US Android Market

submitted by /u/Craznk [link] [comments]