본문으로 건너뛰기
AI NEWS JOURNAL
Hacker News Show HNAI NEWS / JOURNAL

Show HN: Triton-ng – Write Nvidia Triton inference back ends in Rust

Article URL: https://github.com/rdcm/triton-ng Comments URL: https://news.ycombinator.com/item?id=50532862 Points: 1 # Comments: 0

번역 서비스가 일시 중지되어 원문을 표시합니다.

A Rust SDK for writing custom backends for NVIDIA Triton Inference Server. Backends run as shared libraries inside Triton. The project also includes an async gRPC client for calling a running Triton server. Targets Triton 25.09 / backend API 1.19 and Rust 1.99. The API checks compatibility at backend startup. The examples below describe the 0.4.0 API. See CHANGELOG for release boundaries. Getting started [lib] crate-type = ["cdylib"] [dependencies] triton-ng = "0.4.0" Implement one hook to call another model and return its output. This complete backend is also examples/minimal-backend, the minimal backend example (the separate batching-strategy example is introduced below): use triton_ng::{Context, CustomBackend, DataType, Error, InferenceRequest, Request, Response}; struct MyBackend; #[triton_ng::backend] impl CustomBackend for MyBackend { async fn model_instance_execute(ctx: Context, requests: Vec) -> Result<(), Error> { for request in requests { let data = request.get_input("INPUT")?.as_bytes()?; let mut inference = InferenceRequest::new(ctx.server(), "minimal_backend_target", -1)?; inference.add_input_with_data("INPUT0", DataType::Fp32, &[1, 10], data)?; inference.add_requested_output("OUTPUT0")?; let result = match ctx.server().infer_async(inference).await { Ok(result) => result, Err(e) => { request.respond_error_with_code(e.code(), &e.to_string())?; continue; } }; let mut response = Response::new(request)?; response .create_output("OUTPUT", DataType::Fp32, &[1, 10])? .write_fp32_vec(&result.output("OUTPUT0")?.as_fp32_vec()?)?; response.send()?; } Ok(()) } } gRPC client Call the custom backend from your application using the gRPC client: use triton_ng_client::{InferInput, InferOptions, TritonClient, TritonClientConfig}; type ClientError = Box; async fn call_example() -> Result<(), ClientError> { let client = TritonClient::new(TritonClientConfig::new("http://localhost:8001")).await?; let result = client.infer( "minimal_backend", None, [InferInput::fp32("INPUT", vec![1, 10], vec![1.0; 10])], ["OUTPUT"], InferOptions::default(), ).await?; for output in result.outputs { println!("{}: {:?}", output.name, output.data); } Ok(()) } The examples below are independent functions or hooks; import their SDK types from triton_ng. Combine state fields when using several examples in one backend. make build compile-checks the Rust snippets in this README and every guide. Compose models with BLS Forward an input to another model without copying it: async fn forward_input(ctx: &Context, request: &Request) -> Result { let shared = request.get_input("INPUT")?.share()?; let mut call = InferenceRequest::new(ctx.server(), "minimal_backend_target", -1)?; call.add_input_with_buffer("INPUT0", DataType::Fp32, &[1, 10], shared)?; call.add_requested_output("OUTPUT0")?; Ok(ctx.server().infer_async(call).await?) } To replace an input prepared by another helper and request only the embedding: fn replace_audio(call: &mut InferenceRequest, samples: i64, audio: Vec) -> Result<(), Error> { call.remove_input("AUDIO")?; call.add_input_with_data("AUDIO", DataType::Fp32, &[1, samples], audio)?; call.remove_all_requested_outputs()?; call.add_requested_output("EMBEDDING")?; Ok(()) } Request editing and buffer lifetimes. Streaming Return tokens as they become available. Configure the model with model_transaction_policy { decoupled: true }: fn send_tokens(request: Request) -> Result<(), Error> { let mut sender = request.into_sender()?; for token in [10, 20, 30] { let mut response = sender.response()?; response.create_output("TOKEN", DataType::Int32, &[1])?.write_i32_vec(&[token])?; response.send()?; } sender.finish()?; Ok(()) } Call a streaming model from another backend with Server::infer_stream, or from an application with TritonClient::infer_stream. See streaming and cancellation for complete examples. Metrics and response statistics Count completed requests on Triton's Prometheus endpoint. Create the counter once in model initialization and call count_request from execute: fn model_initialize(model: &Model) -> Result<(), Error> { let name = model.name()?; let counter = CounterFamily::new("minimal_requests_total", "Requests handled")? .counter(&[("model", name.as_str())])?; model.set_state(counter)?; Ok(()) } fn count_request(ctx: &Context) -> Result<(), Error> { ctx.state::()?.increment(1.0)?; Ok(()) } Request and batch statistics are automatic. Enable per-response statistics when needed: fn answer_with_statistics(request: Request) -> Result<(), Error> { let mut response = Response::new(request)?; response.create_output("OUTPUT", DataType::Fp32, &[1, 10])?.write_fp32_vec(&[0.0; 10])?; response.enable_statistics(); response.send()?; Ok(()) } Metric types, error responses, and statistics. Tracing and metadata Add a trace activity and connect a BLS call to its client request: async fn traced_call(ctx: &Context, request: &Request) -> Result { request.trace()?.report_activity("decode")?; let mut call = InferenceRequest::new(ctx.server(), "minimal_backend_target", -1)?; call.set_parent_trace(request)?; if let Some(id) = request.id()? { call.set_id(&id)?; } call.set_parameter("language", "en")?; call.add_input_with_data("INPUT0", DataType::Fp32, &[1, 10], request.get_input("INPUT")?.as_bytes()?)?; Ok(ctx.server().infer_async(call).await?) } Request IDs, sequence flags, priority, timeouts, and tracing. Memory buffers Choose memory for each output, then take the buffers back for reuse. This example takes a prepared request for a model with three FP32 outputs of shape [1, 10]: async fn reuse_outputs( ctx: &Context, mut call: InferenceRequest, device_id: i32, ) -> Result<(RamBuffer, PinnedRamBuffer, GpuBuffer), Error> { call.add_requested_output_into("CPU_OUTPUT", ctx.memory().ram(40)?)?; call.add_requested_output_into("PINNED_OUTPUT", ctx.memory().pinned(40)?)?; call.add_requested_output_into("GPU_OUTPUT", ctx.memory().gpu(device_id, 40)?)?; let mut result = ctx.server().infer_async(call).await?; Ok(( result.take_ram("CPU_OUTPUT")?, result.take_pinned("PINNED_OUTPUT")?, result.take_gpu("GPU_OUTPUT")?, )) } Use any one of these buffer types, or combine them as above. Pinned and GPU allocations require Triton's corresponding memory pools. A GpuBuffer can become the next BLS call's input without a host copy; CPU access to its data requires your own CUDA code. Report memory held by your model so clients can inspect it in model statistics: fn model_initialize(model: &Model) -> Result<(), Error> { let allocation = model.memory_manager()?.pinned(4096)?; model.set_state(allocation)?; model.report_memory_usage(&[BufferAttributes::new(MemoryType::CpuPinned, 4096)?])?; Ok(()) } For per-instance allocations use ModelInstance::report_memory_usage. Memory ownership, GPU chaining, and caching. Configure backend and instances Keep a setting shared by every model of your backend: fn initialize(backend: &Backend) -> Result<(), Error> { let language = backend.config()?.parameter_or("language", String::from("en"))?; backend.set_state(language)?; backend.set_execution_policy(ExecutionPolicy::Blocking)?; let (_, directory) = backend.artifacts()?; println!("Backend files: {}", directory.display()); Ok(()) } fn language(ctx: &Context) -> Result<&str, Error> { Ok(ctx.backend_state::()?.as_str()) } Inspect an instance and save its host policy for later input reads: fn model_instance_initialize(instance: &ModelInstance) -> Result<(), Error> { println!("{} passive={} profiles={:?} secondary={:?}", instance.name()?, instance.is_passive()?, instance.profile_names()?, instance.secondary_devices()?); instance.set_state(instance.host_policy()?)?; Ok(()) } Backend configuration, execution policies, and instance settings. Host-policy input data Read the input storage selected for this instance: fn read_policy(ctx: &Context, request: &Request) -> Result, Error> { let policy = ctx.instance_state::()?; request.get_input("INPUT")?.as_bytes_for_host_policy(policy.name()) } Supply different input bytes for a BLS target's host policy: fn policy_call(ctx: &Context, default: Vec, local: &[u8]) -> Result { let mut call = InferenceRequest::new(ctx.server(), "minimal_backend_target", -1)?; call.add_input_with_data("INPUT0", DataType::Fp32, &[1, 10], default)?; call.append_input_data_with_host_policy("INPUT0", local, "cpu")?; Ok(call) } Each input above contains ten FP32 values (40 bytes). A policy without supplied data uses the ordinary input. Policy selection and zero-copy appends. Sequence state and iterative requests Count requests in a sequence. Declare a state tensor with an initial value of zero, read its StateConfig with Model::sequence_states, then update it on each request: fn increment(mut request: Request, config: &StateConfig) -> Result<(), Error> { let previous = request.get_input(config.input_name())?.as_fp32_vec()?; let previous = previous.first().copied().ok_or("state has no counter")?; let mut state = State::new(&mut request, config, &[1])?; state.write_bytes(&(previous + 1.0).to_ne_bytes())?; state.update()?; let mut response = Response::new(request)?; response.create_output("OUTPUT", DataType::Fp32, &[1])?.write_fp32_vec(&[previous + 1.0])?; response.send()?; Ok(()) } This example uses max_batch_size: 0 and state dimensions [1]. Complete sequence-state configuration. Return a partial result and schedule another step of a decoupled iterative sequence: fn next_step(request: Request) -> Result<(), Error> { let mut sender = request.into_sender()?; let mut response = sender.response()?; response.create_output("STEP", DataType::Int32, &[1])?.write_i32_vec(&[1])?; response.send()?; sender.reschedule()?; Ok(()) } Triton r25.09 allows only one reschedule per request; finish the response on the second step. Two-step execution and model configuration. Custom batching Group adjacent requests with the same batch_key parameter: struct ByKey; struct Batch { started: bool, key: Option } #[triton_ng::batch_strategy] impl BatchStrategy for ByKey { type Batch = Batch; fn initialize(_model: &Model) -> Result { Ok(Self) } fn batch(&self) -> Batch { Batch { started: false, key: None } } fn include(&self, batch: &mut Batch, request: &RequestView<'_>) -> Result { let key = match request.parameter("batch_key")? { Some(Parameter::String(value)) => Some(value), _ => None, }; if !batch.started { batch.started = true; batch.key = key; return Ok(true); } Ok(batch.key == key) } } Build this as a separate library and set TRITON_BATCH_STRATEGY_PATH in the model config. Runnable example · Queue behavior and safe model unload. Comparison with python_backend Against python_backend r25.09: Full comparison, including feature gaps and cases where Python is the better choice. Benchmarks The recorded benchmark compares five small BLS workloads on the same Triton 25.09 server, two CPU instances per backend, with a Ryzen 9 9950X3D. At concurrency 64: These are the recorded measurements, not a promise for large GPU models; heavy inference reduces the backend's share of total time. Streaming latency in these results is time to first response. Method, environment, all concurrency levels, and latency tables are in the benchmark guide. Documentation Start with the topics you need; the guides and this README are also included in the Rust API documentation. Development Install Rust, Docker, the NVIDIA Container Toolkit, and a compatible NVIDIA driver. The test suite also requires cargo-nextest. From the repository root: make build make download-model make docker-env-up make tests make docker-env-down Wait until Triton reports the models ready before running the tests. After changing a backend, rebuild and restart the service. See the Makefile for all development commands.

자료 출처

원문 읽기 ↗

관련 주제

← 전체 소식 이어 읽기