跳到正文
AI NEWS JOURNAL
VercelAI NEWS / JOURNAL

AI Gateway now supports Private Inference

Vercel's AI Gateway now enables enterprises to integrate private, self-hosted, or prerelease models through a unified interface, offering consistent authentication, SDKs, APIs, and observability. Users can switch between private and catalog models using the same endpoints, keep models scoped to their team with encrypted credentials and optional Zero Data Retention, connect to coding agents like Claude Code and Cursor, monitor traffic alongside public models, configure fallbacks to catalog models, and manage providers via CLI. Private model requests are logged alongside catalog requests for unified observability.

翻译服务暂未运行,当前显示原文。

Private Inference connects your own model endpoints to AI Gateway. Enterprises running fine-tuned, self-hosted, or prerelease models used to need separate keys, client code, and logs for them. Those models can now run through the same authentication, SDKs and APIs, and observability as catalog models, so every model your team uses is managed in one place. Use Private Inference to: Switch between private and catalog models: Call private models through Chat Completions, Responses, or Messages. AI Gateway translates each request to your endpoint's format, so switching only changes the model ID. Keep models scoped to your team: Only your Vercel team can view or call private models, and they aren't listed in the public catalog. Credentials are stored encrypted, and you can enable Zero Data Retention per provider. Connect coding agents: Use private models in Claude Code, Codex, Cursor, and other coding agents connected to AI Gateway. Monitor traffic together: View private model requests alongside catalog traffic in logs, trace drains, and custom reporting. Observability charts break down requests, tokens, and time to first token by private model. Configure fallbacks: Optionally fall back to a catalog model if your private endpoint fails. To get started, open the Private Inference page and add your endpoint as a private provider. Enter its base URL, select its API format, and add its credentials. Then add the models it serves. You can also register the provider and model with the Vercel CLI, version 65.16.0 or later: # Connect your endpoint as a private providervercel ai-gateway private-inference providers create acme-vllm \ --base-url https://vllm.example.com/v1 # Expose the endpoint's model as private/support-fine-tunevercel ai-gateway private-inference models create support-fine-tune \ --provider acme-vllm \ --api-model acme/support-llama-ft Call the model using its private/ ID: import { streamText } from 'ai'; const result = streamText({ model: 'private/support-fine-tune', prompt: 'Draft a reply to a customer who was charged twice.', providerOptions: { gateway: { // Optional fallback to a catalog model models: ['anthropic/claude-sonnet-5.5'], }, },}); for await (const text of result.textStream) { process.stdout.write(text);} Private model requests appear in AI Gateway logs alongside requests to catalog models. Read the Private Inference documentation for supported model types, requirements, and billing details.

资料来源

阅读原文 ↗

相关主题

← 继续阅读全部动态