跳到正文
AI NEWS JOURNAL
Hacker News Show HNAI NEWS / JOURNAL

Show HN: Agent Doctor – let domain experts tune your agent without code changes

Hi HN , I'm Abhinav Prakash and I built Agent Doctor. It lets a domain expert (lawyer, doctor, banker) change steps in an AI agent and re-run it on your own servers, without code changes or redeploys.I would like to learn: how does your domain expert's feedback reach your agent today? I'd also really value a review of what I built, what's missing and what's wrong.Thanks ! Comments URL: https://news.ycombinator.com/item?id=50598128 Points: 4 # Comments: 0

翻译服务暂未运行,当前显示原文。

Two lines of SDK code capture every LLM call, vector lookup and tool call. Anyone on your team can then override a prompt, model or temperature from the dashboard and re-run it on your own servers, as many times as it takes. Ship the winner when it's proven. Override any prompt and re-run the whole pipeline on your servers. A cardiologist with no coding background spots a wrong diagnosis, rewrites the faulty prompt from the dashboard, and re-runs the agent until the answer is right. The developer ships the proven prompt. Developers build the pipeline and domain experts tune it. An engineer can wire up the agent, but they can't judge a sound medical diagnosis or a legally airtight clause. Agent Doctor gives the people who can a dashboard to iterate on prompts and parameters, while the developer keeps ownership of the code. Builds the pipeline Writes the agent code, connects to databases, sets up tools, deploys to production. Adds two lines & a webhook Drops in the SDK and one webhook handler for re-runs. Back to building features. Reviews & ships When the domain expert finds a winning prompt, the dev reviews it and ships it with the next release. Sees every step visually Opens the dashboard, sees the full pipeline trace. Every prompt, every output, every decision the agent made. Tweaks prompts & parameters Spots a bad prompt, rewrites it in plain English. Adjusts the temperature. Hits "Run". No code, no deploys. Iterates until it's right Compares runs side-by-side, uses Probe to debug, runs consistency checks. All from the dashboard. Your existing code. Unchanged. Two lines to capture. One webhook handler to re-run. Agent Doctor hooks into LLM SDKs, vector databases and tool calls at the library level, so your agent logic stays exactly as-is. # Add these two lines, that's it import agent_doctor agent_doctor.init(api_key="sk-proj-...") # Your existing agent logic, unchanged # LLM calls, vector DB queries and tool calls are auto-captured def handle_ticket(query, customer_id): # LLM call → auto-captured intent = openai.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": query}] ) # Vector DB lookup → auto-captured docs = pinecone_index.query(vector=embed(query), top_k=5) # Tool call → auto-captured policy = check_refund_policy(customer_id) # LLM call → auto-captured response = openai.chat.completions.create( model="gpt-4o", messages=[{"role": "system", "content": f"Docs: {docs} Policy: {policy}"}, {"role": "user", "content": query}] ) return response # Your webhook handler. payload is the run Agent Doctor sends. def on_run(payload): with agent_doctor.run_context(run_id=payload.run_id, overrides=payload.overrides): result = handle_ticket(payload.query, payload.customer_id) # Every step auto-captured. Overrides auto-injected. Done. 01The dashboard sends the overrideA new prompt, model or temperature for one node, posted to your webhook. 02Your servers run the agentrun_context injects the override at runtime. No code change, no redeploy, full access to your private data and tools. 03Only the trace comes backInputs, outputs and timings for each step land in the dashboard to compare. Two lines in, every step of your pipeline is in the dashboard. Probe finds the broken node by asking you a few questions. Tell it what the agent should have said and what it said instead. It walks the trace with you, asking three or four plain-English questions, and names the exact step that caused the failure. No logs, no code, no debugging skills needed. A chat, so anyone on the team can run it Checks the pipeline by halves, so it needs only a few questions Highlights the broken node in the trace Hands the fix straight to the prompt optimizer Consistency checks find the step that answers differently every time. Run your pipeline ten times with identical input. Per-node consistency scores show exactly which step is non-deterministic, and an AI report tells you why. Per-node consistency scores and decision divergence AI root cause report with a suggested fix Apply the fix, re-test, and verify from the same screen The optimizer writes, tests and ranks better prompts for you. Pick a node and define what "good" looks like. The optimizer generates prompt variants, runs them against your examples, and ranks the results. It starts with GEPA, a genetic strategy that evolves prompts across generations, with more strategies on the way. Evolves prompts across generations and keeps the fittest Rephrases, expands, compresses and recombines winning traits Ranks every variant by its score on your examples, with token cost weighed in Shows the winning prompt side by side with the original prompt Why not just use a regular observability tool? Most LLM tracing platforms run experiments on their own cloud, so your agent can't reach your private databases, RAG, or internal tools during a test. They also keep prompt iteration locked behind code, leaving your domain experts out. Common questions What leaves my servers? How much code do I have to change? Which languages and frameworks are supported? Can non-technical teammates actually use it? What does it cost? Test every agent fix. No code changes. No redeploys. Install the SDK, open the dashboard, and let the whole team work on the agent. Built with love in Khunti, Kudlum, Jamshedpur, Bangalore, Delhi, India

资料来源

阅读原文 ↗

相关主题

← 继续阅读全部动态