Skip to content
AI NEWS JOURNAL
AI, IN PERSPECTIVE

Search AI news

Search headlines, summaries and articles to connect the dots.

Newest first

JigSLO: Solving the Resource Puzzle for Efficient Multi-LLM Serving

arXiv:2509.06261v3 Announce Type: replace Abstract: GPU spatial sharing co-locates instances with complementary demands, such as compute-bound prefill and memory-bound decode, promising cost-efficient multi-LLM serving. Prior systems partition compute and coordinate shared memory capacity, but do not model bandwidth contention among co-located instances of heterogeneous LLMs. This contention makes each instance's latency depend on the memory-access patterns of its co-runners, even when its own resource allocation remains unchanged. Drawing on extensive measurements, we find that the latency impact of bandwidth contention can be captured by a time-shared memory-access model with contended and uncontended periods, so the overall latency can be analytically estimated as a time-weighted average of the latency estimates for these periods. We propose JigSLO, a contention-aware multi-LLM serving system that manages memory resources and schedules each model's prefill and decode phases as separate instances, treating their co-location as a placement decision. Its contention-aware predictor estimates per-instance latency under any co-location from standalone profiles alone. Guided by these predictions, its placement algorithm jointly selects each instance's tensor parallelism, compute share, and batch configuration, pairing complementary memory-access patterns to maximize throughput within each model's latency targets. JigSLO's shared resource manager complements this on the capacity side, sharing weights, pooling KV cache, and transferring it between prefill and decode instances. Across three datasets covering chat, code generation, and long-context summarization, JigSLO outperforms state-of-the-art baselines by $2.3\times$ in SLO attainment and $1.2\times$ in throughput on average, matching their best attainment with half the GPUs on long-context workloads.

KV-LTM: Persistent Per-Session KV Caches as Long-Term Memory for Personal Assistants

arXiv:2611.21998v1 Announce Type: new Abstract: Personal assistants accumulate months of conversations, but today's long-term memory stacks put derivation or selection between a question and its evidence. In a system-level comparison on LongMemEval, two commercial assistants scored well below the same model reading the history offline [1], consistent with, but not isolating, content loss between write and read. We propose KV-LTM: a memory tier storing each session's raw text plus its KV cache under a fixed reader model, answering by restoring cached sessions and appending the question. KV-LTM is complementary to, not a replacement for, retrieval and indexing. Text is the memory of record; the cache is a recompute-free prefill accelerator, degrading gracefully to text. We specify the design: storage record, budget-derived retention, append-only read path, and bounded aggregation. We size it analytically: across pinned BF16 full-attention (GQA) and latent-cached MLA configurations, per-token state spans approximately 31K-328K bytes (the pinned KDA+MLA hybrid differs: approximately 8.1K bytes/token plus a fixed per-session recurrent term), giving approximately 0.34-3.6 TB per user-year at a declared 30K-token/day scenario, against approximately 49 MB of raw text; per-query cost is modelled as regime-conditional transfer, attention, decode, aggregation, and compute. We define an evaluation protocol needing no persistent-cache implementation for logical QA evaluation: per-session prefill emulates valid cache reuse under the same model/input-state contract and ideal arithmetic, with named arms, controlled baselines, and a preregistered reader-model gate; a separate minimal real-cache replay tests numerical equivalence and measures the cost axis, with outcomes not assumed. KV-LTM trades storage for avoided re-prefill compute and write-time fidelity, within a declared retention budget.

Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference

Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo for distributed inference, with multi-node and disaggregated serving, KV-aware routing, and automatic fault recovery. Serve large models across many nodes without building the orchestration layer yourself.

MInfoVisQA: A Multilingual Benchmark for Information-Rich Visual Question Answering

arXiv:2611.21644v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) excel in vision-language tasks; yet, whether they maintain consistent understanding of information-rich visuals (InfoVis) when the surface language changes remains unclear. To quantify this capability, we introduce MInfoVisQA, a parallel benchmark with 8,960 QA configurations derived from 128 seed instances across 24 languages. We construct seed-matched, language-parallel visuals and QA pairs by disentangling numerical-structural data from text and programmatically synchronizing translated labels. We evaluate 25 MLLMs under two settings: Language-Aligned QA (LQA, matching visual-text languages) and Cross-Language QA (XQA, fixed QA language with varying visual languages). The results show that aggregate accuracy masks substantial robustness gaps: two models with comparable mean LQA accuracy (80.4\%) exhibit performance spreads of 7.8 and 20.3 percentage points across languages. Moreover, all 25 models score lower with non-English visuals under fixed English QA, underscoring the need for cross-language sensitivity metrics.

OneTrust’s Fall ’26 Release expands AI visibility, privacy and third-party risk tools

Governance platform provider OneTrust LLC today announced its Fall ’26 Release, a set of platform updates said to bring governance into the workflows where business decisions are made. Building on OneTrust CORIE, a shared intelligence layer for governing artificial intelligence agents introduced at the company’s TrustWeek conference in September, the release covers AI visibility, privacy, […] The post OneTrust’s Fall ’26 Release expands AI visibility, privacy and third-party risk tools appeared first on SiliconANGLE.

Parallel web search is free on AI Gateway and eve through January 31

Parallel web search in fast mode is free on AI Gateway until January 31, 2027, and is now the default for eve agents. It works with any AI Gateway model without a separate API key, returning links and excerpts with filtering options by website or date. The feature is automatically used in eve's web_search tool when models are served via AI Gateway. Developers can enable it via the AI SDK using gateway.tools.parallelSearch({ mode: 'fast' }). Fast mode includes Zero Data Retention by default; other modes are billed at standard rates.

Reason over time-series data with OpenTSLM on Amazon SageMaker JumpStart

OpenTSLM, an open source family of time-series language models, is now available in Amazon SageMaker JumpStart. See how Nexthink and Aionic Labs apply it to proactive IT support, then deploy OpenTSLM to a real-time endpoint and fine-tune it on your own data with TimeNet and a SageMaker training job.

Reducing token costs and latency in Amazon Bedrock AgentCore JSON workloads

As production agents on Amazon Bedrock AgentCore process multi-megabyte JSON payloads, token costs and latency climb. This post shares three composable strategies: offloading heavy artifacts to Amazon S3, on-demand result paging, and dictionary compaction. Together they cut per-turn token use by up to 71% and latency by up to 80%.

Responsible AI for agentic AI: A phased roadmap for platform and application teams

Enterprises adopting agentic AI at scale need platforms that embed responsible AI from the start. This post gives platform and application teams a three-phase roadmap (governance foundation, pilot deployment, and evaluation and hardening) that maps the AWS Responsible AI Lens pillars to each phase, with guidance on when to revisit each one.

Secure OAuth patterns for Snowflake Cortex AI through Amazon Bedrock AgentCore

When an AI agent on Amazon Bedrock AgentCore calls Snowflake Cortex AI over MCP, how does it authenticate? This post walks through three OAuth patterns we tested on AgentCore Identity: Snowflake OAuth, three-legged OAuth with a corporate IdP, and two-legged OAuth, so you can match your organization's identity, audit, and compliance needs.

Show HN: Sip pacer that helps you drink slower

It's a new feature I've added to my app Pour Decision.The idea is:1. Set an interval on your phone and slip it in your pocket2. The phone vibrates in your pocket on every interval3. Condition yourself to have sips when you feel the vibrationBenefits:- You end up drinking way less in a night than you normally would- You don't feel like you're moderating cus you're never without a drink- It's discreet so people you're drinking with won't know you're doing thisThis is sort of like a HIIT-timer, but to preserve battery I took another approach. It works by pre-scheduling on-device notifications when you start the timer. iOS allows apps to schedule up to 64 notifications. If you were to set say 5-minute intervals, that covers the next 320 minutes - more than 5 hours. The phone uses no battery in between intervals, and only wakes up the device once every 5 minutes, so battery drain is negligible. Comments URL: https://news.ycombinator.com/item?id=50542566 Points: 2 # Comments: 0

Show HN: Spillway – HTTP API load testing on macOS

I've used JMeter for years — it's free and has a truly amazing amount of functionality - but it doesn't really feel like a Mac app. There's also a bunch of other more modern features that I wish it had.Spillway is scratch-my-own-itch app... a culmination of things I've wished for over the years: native macOS app, visual test plans, easy-to-use variable substitution, built-in support for modern OAuth patterns, ability to compare previous runs graphically, and no wild per-user/request pricing.Genuinely pleased with how well it has turned out. Happy to field any questions or feature requests Comments URL: https://news.ycombinator.com/item?id=50540777 Points: 1 # Comments: 0