跳到正文
AI NEWS JOURNAL
NVIDIA DeveloperAI NEWS / JOURNAL

Build Tool-Calling Parsers in Rust with AI Agents

A tool call can arrive in pieces, an argument can be cut short, or model-specific markup can leak into the answer. A frontend parser turns that output, received...

翻译服务暂未运行,当前显示原文。

AI-Generated Summary frontend-crates provides reusable Rust components that parse tool calls from large language model outputs into structured events for applications. The parsers handle malformed, incomplete, and streaming inputs by following documented recovery rules rather than guessing model intent, preserving valid data without leaking model-specific markup. A shared Conformance Corpus pairs model-specific inputs with expected GOLDEN outputs, enabling behavioral tests such as UNIFIED.* cases to be reused across model families including Beam, GPT-OSS, Muse Spark, and NVIDIA Nemotron 3. Developers create new parsers by defining a GOLDEN test set adapted from UNIFIED.* cases, implementing the parser against the model's grammar, and validating results with the local conformance report. Guided decoding support constrains generation to the model's markers and argument schema to reduce malformed calls, while parser recovery checks remain necessary for truncation and semantic errors. An example demonstrates using the NVIDIA Nemotron parser from the dynamo-parsers-v2 crate to process streamed chunks and produce ordered reasoning and tool-call events. Next Steps Explore the frontend-crates repository to access parser source code, fixtures, and implementation instructions. Review the Parser Conformance Table to compare parser behavior across model families. Follow the latest repository instructions for adding a day-0 tool-calling parser. A tool call can arrive in pieces, an argument can be cut short, or model-specific markup can leak into the answer. A frontend parser turns that output, received as token IDs or decoded text, into visible text, reasoning, and tool calls an application can use. In this post, you’ll learn how tool-call parsing works and use an AI agent to build a parser for a new model. You’ll start with a weather request and a human-reviewed test set, using frontend-crates, the reusable Rust components behind NVIDIA Dynamo tool-calling parsers. Suppose you ask for the weather in Paris. The model may generate reasoning followed by a get_weather call with a city argument. The frontend prepares and tokenizes the prompt, then an inference engine runs the LLM to generate tokens. The parser consumes token IDs or decoded text and separates reasoning, visible text, and tool calls in order. A protocol adapter turns those events into an OpenAI-compatible response, using the same parsing path for batch JSON and streaming updates. The tricky part is recognizing where each piece starts and ends. A call can be malformed or incomplete, a marker can appear inside an argument, or ordinary text can surround the call. The parser has to handle these boundaries without losing data, duplicating calls, or leaking model-specific markup. Returning to the weather request: the model might stop halfway through a city name or call a tool that wasn’t offered. Either can disrupt the agent’s next action. The following NVIDIA Nemotron examples show these two failures. Syntax error. The argument and tool-call frame never close, so no complete call can be parsed. Semantic error. The frame is valid, but the named tool is absent from the request. These failures require different handling. If the model names an unavailable tool, the parser reports the call it received, and the serving layer validates tool availability, authorization, schema, and call policy. Changing the tool name would hide what the model actually generated. An incomplete call is harder to judge while the stream is open. The rest of the city name may arrive in the next chunk, and a marker can split across chunks, too. Emit too early and syntax leaks into the answer; wait too long and ordinary text gets delayed. The parser needs consistent rules for what to emit, what to buffer, and how to handle the end of the stream. A parser can encounter an unbounded range of inputs. With invalid input, it must follow documented recovery rules rather than guess the model’s intent. Even a syntactically valid call that conforms to the schema can be the wrong choice for the user’s request. The parser needs a consistent grammar and recovery policy so it can preserve valid data without inventing calls or leaking model-specific markup. To check those decisions, frontend-crates pairs model outputs with expected results in a shared Conformance Corpus. Using the Conformance Corpus Use the Conformance Corpus to check how applicable parsers handle behavior first observed in one model. Its language-neutral YAML fixtures pair fixed input with expected output, so Rust parsers and Python references can run the same cases. A regression test for leaked markup, duplicate calls, or lost arguments can cover other families when adapted to their grammars. For a new parser, start with all UNIFIED.* cases across model families. They cover complete, malformed, truncated, multiple, and interleaved calls, plus streaming boundaries and end-of-input behavior. Adapt inputs to your model’s grammar and request settings, add model-specific edge cases, and explain exclusions. You can’t test every possible output, but you can reuse cases from other formats. Table 1 shows the UNIFIED.1-tool_only case from the Parser Conformance Table. In this example, the GOLDEN test set pairs four model-specific inputs with the same expected result: get_weather with city set to Paris. The markers differ, but each case tests the same behavior. Adapting the input format makes the same behavioral test reusable across models. Tutorial: Create a new parser Now you can create a parser for a model that frontend-crates doesn’t support yet. Ask your AI agent to read this post and work through the tutorial, starting with the prompt in step 1. Your model’s specification defines the grammar; the shared cases provide behaviors to test. You’ll need a frontend-crates checkout, Rust and Cargo, and the conformance dependencies. The parser exercise runs on a CPU using fixed test fixtures, so you don’t need a GPU. End-to-end testing also requires a working Dynamo runtime, discovery services, and hardware for your model. When you’re ready to test end-to-end, launch the tests with Dynamo runtime on the actual model and GPU(s). The workflow in Figure 2 starts with the model’s grammar, recovery rules, and reviewed GOLDEN expectations. The agent writes or fixes the parser, runs tests, and uses failures to guide the next change. If it proposes changing the expected behavior, review that decision before accepting it. Test your parser end to end with real models before claiming deployment support. Plan time to allocate a machine, load the model, initialize the runtime, and run your own test cases. Turn failures into unit tests for the agent, then rerun the end-to-end test to check the fix. 1. Start with a prompt Give the agent this prompt: In the frontend-crates checkout, follow .agents/skills/tool-parser-generator/SKILL.md to create a v2 Unified parser for [MODEL_NAME]. Research its grammar and prepare the GOLDEN test set, adapting applicable UNIFIED.* cases. Pause for human review. After approval, implement, validate, and provide the local conformance report URL for human inspection. 2. Define the GOLDEN test set The GOLDEN test set defines the behavior your parser will be tested against. Use the prompt in step 1 to have the agent generate an initial test set in conformance/fixtures-unified-v2/families//inputs_and_golden.yaml, independently of parser results. Each case pairs an input with its expected output, called GOLDEN. If those expectations are wrong, the parser can pass the tests and still behave incorrectly. The following example pairs an input with its expected GOLDEN output. It omits request definitions, tools, initialization, and chunk settings; the existing nemotron_v3/inputs_and_golden.yaml shows the full format. family: nemotron_v3 cases: UNIFIED.tool_only.nemotron_v3: input: |- Paris golden: - kind: tool_call name: get_weather arguments: {city: Paris} Ask the agent for an adversarial review, then go through the set manually, case by case. The initial test set may contain errors because expected behavior depends on context the agent may not have. Review it against the following sources and requirements: The model format and API specification. The reference implementation. Other parsers, including vLLM and SGLang. The grammar, tool schema, and request settings. Relevant forum discussions and existing tickets. Caller needs and recovery costs in latency and complexity. Where sources disagree, you may need to make a judgment call. Write down the recovery rule and why you chose it, and document the reason for any GOLDEN change. Expectations should follow that decision, not the parser’s output. 3. Implement the parser and inspect the results After reviewing the GOLDEN test set, ask the agent to continue: build the parser, run all applicable UNIFIED.* cases and shared regression tests, and fix the failures. Have it recapture results, generate conformance/CONFORMANCE_v2.html locally, and provide the local report URL so you can inspect its work. Open that URL in your browser. The local report uses the published Parser Conformance Table‘s interface with your local results. Inspect each case’s input, request settings, GOLDEN, and recorded output. Investigate regressions and missing captures. Green means the output matches GOLDEN, so the table depends on the expectations you reviewed. Figure 3 shows the report interface. Guided-decoding support The agent’s implementation step also covers guided decoding, which can reduce malformed calls by constraining generation to the model’s markers and argument schema. Ask your agent to follow “Add and validate structural tags” in the parser README and “Implement and validate” in the agent skill. These instructions cover builder registration, grammar tests, and backend qualification. Guided decoding can’t ensure that the model chooses the right city or prevent truncation, so keep the parser’s recovery checks. Common gotchas: speed, streaming, and memory Passing correctness tests is only a starting point for an agent-generated parser. The implementation may still be slow or use too much memory. Try plain text, tool calls, long arguments, and malformed input with large and tiny chunks. Measure throughput, time to the first event, and memory use on the same build and hardware. Double the input length to spot code that keeps rescanning or copying what it already read. With streaming, watch what the caller receives as each chunk arrives. A parser can return the right final result and still make the caller wait until the end. Replay short fixtures at every split point, then try one-character and mixed chunks, including splits inside markers, escaped quotes, and arguments. Check that events arrive in order and arguments stay intact. If events are queued, test a slow consumer and cancellation too. Also try cutting the stream off halfway through a marker or argument. Check that recovery follows the documented rule without leaking markup, duplicating calls, or losing valid text. The parser should handle the input it received without guessing the missing pieces. Test text containing examples of the model’s format as well as tool calls against the documented grammar and quote-handling policy. The current v2 default treats recognized markers as structural even inside quoted prose, so quotation marks alone do not keep a complete tool-call example in ordinary text. Record the configured policy and test its expected behavior. Give the agent this prompt to check for common gotchas: Check this parser for common gotchas. Benchmark speed and memory with short, long, and malformed inputs using different chunk sizes. Look for repeated scanning, copying, or growing buffers. Verify that output arrives incrementally and stays correct across chunk boundaries and truncation. Where queues exist, test a slow consumer, backpressure, and cancellation. Check literal format examples and real tool calls against the documented grammar and configured quote-handling policy. Report measurements, reproducible failures, and anything you could not verify. How to use the parser crate in your own Rust project Choose a parser whose grammar matches your model. This example uses the existing NVIDIA Nemotron parser and splits both a marker and the city name across chunks. This example was tested with Rust 1.96.1 and dynamo-parsers-v2 0.10.3. Run cargo new parser-demo, then cd parser-demo. Add the following entries under [dependencies] in Cargo.toml: anyhow = "1" dynamo-parsers-v2 = "=0.10.3" serde_json = "1" Replace src/main.rs with: use dynamo_parsers_v2::{ Tool, UnifiedParserInit, UnifiedParserOutput, create_unified_parser_for_family, }; use serde_json::json; fn main() -> anyhow::Result<()> { let tools = [Tool { name: "get_weather".into(), description: None, parameters: json!({"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}), strict: None, }]; let mut parser = create_unified_parser_for_family("nemotron_v3", &tools)?; parser.initialize_request(UnifiedParserInit::default())?; let mut output = UnifiedParserOutput::default(); for chunk in [ "Check the forecast.Par", "is", ] { let mut delta = UnifiedParserOutput::default(); parser.parse_into(chunk, &mut delta)?; output.append(&mut delta); } output.append(&mut parser.finish()?); println!("{}", serde_json::to_string(&output.assembled())?); Ok(()) } Run cargo generate-lockfile, then cargo run --locked. The result contains reasoning followed by a get_weather call with city set to Paris: [{"kind":"reasoning","text":"Check the forecast."}, {"kind":"tool_call","name":"get_weather","arguments":{"city":"Paris"}}] Keep one initialized parser for each request choice. create_unified_parser_for_family selects the grammar. initialize_request sets the starting state. parse_into consumes each chunk. Finalize once: use finish() for truncation, cancellation, or the end of the transport stream. Use finish_at_eos() only for a backend-confirmed grammar-ending end-of-sequence (EOS) signal. Use assembled() for the ordered results. In a streaming application, forward the events in delta.events after each parse_into call and the events returned by finish(). This example collects the events to print the final output. Before executing tools, validate completed calls for tool availability, authorization, schema, and call policy. Get started To build a parser for your model, start with the source, fixtures, and instructions in frontend-crates and adapt an existing test case. Give your agent the model’s format specification and the repository instructions, then use the Parser Conformance Table to check the parser’s recorded output against GOLDEN. When you open an issue or pull request, include the exact failing input, case identifier, and test results so reviewers can reproduce the behavior. You can use these parsers in your own Rust project or as part of NVIDIA Dynamo, an inference framework that uses frontend-crates. Before you begin, check the latest repository instructions so you and your agent follow the current steps. Acknowledgements Jesse Gu, Alec Flowers, Dong Xu, Qi Wang, and the many partners and proofreaders who helped with this blog and frontend-crates. About the Authors Comments Comments are closed.

资料来源

阅读原文 ↗

相关主题

← 继续阅读全部动态