Skip to content
AI NEWS JOURNAL
AI, IN PERSPECTIVE

Search AI news

Search headlines, summaries and articles to connect the dots.

Newest first

Cut self-hosted LLM cost and latency with the distributed KV caching agent skill

Self-hosting LLMs on GPUs means paying to recompute the same context on every turn. Distributed KV caching with Amazon ElastiCache can cut inference cost by up to 40% and time to first token by up to 80%. This post walks through the distributed KV caching skill in the Agent Toolkit for AWS, which evaluates and sets up the cache from your coding agent.

Matching Architecture to Workload: Comparing RISC-V Parallel Accelerators for Energy-Efficient Embedded Computing

arXiv:2611.25947v1 Announce Type: new Abstract: Embedded computing demands high performance and energy efficiency across workloads with widely different computational characteristics. Choosing the most suitable execution model therefore requires a quantitative understanding of how each model maps to different workload classes. In this paper, we compare three execution models for energy-efficient embedded computing: scalar multicore, stream ing multicore, and vector processing. We use three open-source RISC-V accelerators as representative implementations: the PULP Cluster, the Snitch Cluster, and Spatz, respectively. Their open-source hardware and software stacks enable a controlled comparison based on implementations in the same 12 nm technology and a common benchmark suite. Our results reveal that no single execution model performs best across all workload classes. Spatz delivers the greatest benefits for regular and highly vectorizable compute-intensive applications, achieving up to 4.29x higher performance and 5.17x greater energy efficiency than the other architectures. However, its effectiveness strongly depends on the availability of data-level parallelism and regular computation and memory-access patterns, making it the most sensitive to workload characteristics. The Snitch Cluster performs particularly well on applications with limited parallelism and streamable memory-access patterns, providing up to 5.53x higher performance and 1.45x greater energy efficiency. Conversely, the PULP Cluster is most effective on irregular and control-dominated applications, achieving improvements of up to 5.66x in performance and 2.80x in energy efficiency. Compared with Spatz, however, the greater flexibility offered by PULP and Snitch comes at a non-negligible cost in terms of both area and power.

Announcing Data Annotation Queues

Review logs, annotate data, and build datasets with LangSmith's Data Annotation Queues. Improve LLM app performance from prototype to production.

Cut generative AI costs: Code for rules, models for judgment

The biggest cost lever in a generative AI system is whether you call a model at all. Learn how to decide whether each workflow step needs a model, and map each option to AWS: deterministic code, a constrained decision call, or a right-sized model on Amazon Bedrock.

Reason over time-series data with OpenTSLM on Amazon SageMaker JumpStart

OpenTSLM, an open source family of time-series language models, is now available in Amazon SageMaker JumpStart. See how Nexthink and Aionic Labs apply it to proactive IT support, then deploy OpenTSLM to a real-time endpoint and fine-tune it on your own data with TimeNet and a SageMaker training job.

Show HN: Sip pacer that helps you drink slower

It's a new feature I've added to my app Pour Decision.The idea is:1. Set an interval on your phone and slip it in your pocket2. The phone vibrates in your pocket on every interval3. Condition yourself to have sips when you feel the vibrationBenefits:- You end up drinking way less in a night than you normally would- You don't feel like you're moderating cus you're never without a drink- It's discreet so people you're drinking with won't know you're doing thisThis is sort of like a HIIT-timer, but to preserve battery I took another approach. It works by pre-scheduling on-device notifications when you start the timer. iOS allows apps to schedule up to 64 notifications. If you were to set say 5-minute intervals, that covers the next 320 minutes - more than 5 hours. The phone uses no battery in between intervals, and only wakes up the device once every 5 minutes, so battery drain is negligible. Comments URL: https://news.ycombinator.com/item?id=50542566 Points: 2 # Comments: 0

Benchmarking LLM Inference on Desktop-Scale NVIDIA Grace Blackwell Clusters

arXiv:2611.20289v1 Announce Type: new Abstract: Two NVIDIA DGX Spark GB10 systems connect directly over 200 GbE to serve larger language models and increase inference capacity, but the benefits depend on how they are deployed. We benchmark 17 open-weight checkpoints spanning dense and mixture-of-experts architectures, 8B to 235B parameters and four weight precisions on one and two RDMA-connected systems. Using vLLM, we compare tensor parallelism (TP=2), pipeline parallelism (PP=2) and a two-replica baseline at concurrency 1 to 64, with three independent launches per configuration. TP=2 delivers a median 1.59x peak-throughput gain over one Spark, versus 1.39x for PP=2 and has higher throughput in 93 of 96 matched comparisons with separated 95% Welch intervals. It reduces unloaded inter-token latency to 0.52x to 0.73x of one Spark, whereas PP=2 provides little reduction. TP=2 also achieves the highest goodput under a strict 100 ms inter-token latency objective wherever both strategies are supported and the objective is met. For models fitting on one Spark, TP=2 generally favors low-concurrency serving, while replication offers higher throughput at high concurrency. One Spark uses less GPU energy per output token at matched concurrency. Across three precision ladders, NVFP4 reduces unloaded latency more than adding a second Spark at BF16. NCCL microbenchmarks and comparisons with an earlier TCP campaign suggest that collective communication overhead contributes to the advantage of TP=2. These results provide deployment guidance for desktop-scale inference.

What Do Multilingual Self-Supervised Speech Encoders Know About Sentence Meaning?

arXiv:2611.17686v1 Announce Type: new Abstract: Self-supervised speech models (S3Ms) have been scaled up to thousands of languages and millions of hours of audio, yet little is known about how these models structure semantic knowledge across languages. In this work, we probe sentence-level semantics in massively multilingual S3Ms across 76 languages, using linear maps from frozen speech representations into a frozen text embedding space, evaluated by cross-modal retrieval. Sentence meaning is linearly accessible but highly uneven across languages and resource tiers. Cross-lingual probes can match language-specific probes on languages never seen in probe training, but per-model monolingual decodability does not predict transfer. Within the OmniASR wav2vec family, scaling from 300M--7B disproportionately benefits high-resource languages, and ASR tuning sharply improves semantic decodability while making representations more language-specific. However, semantic content is entangled with acoustics, inflating apparent performance on single-speaker corpora.

Building a self-service LLM gateway with Amazon Bedrock and LiteLLM

Build a self-service LLM gateway with LiteLLM on AWS Fargate that puts Amazon Bedrock foundation models behind an OpenAI-compatible API. Developers get instant access through a single API key, while platform teams gain per-user cost controls, quota isolation, and observability.

Is there a 'right amount' of AI?

Article URL: https://coyotetracks.org/blog/right-amount-of-ai/ Comments URL: https://news.ycombinator.com/item?id=50480659 Points: 32 # Comments: 40

Show HN: Deccy – software engineering as decision management

Hi HN,We built deccy out of frustration from working with coding agents.Although we love the speed that AI has brought to our work, we also want to have control over what we build. While it's not impossible to keep up with the changes AI makes, it is significantly more difficult when compared to writing code ourselves.We realised that what we cared about most was not controlling every line of the implementation, but preserving our intent (we especially hated when agents would wrongly guess what we wanted). We iterated through planning and spec-driven methods, but found that both writing our own plans/specifications and reading AI-generated versions was tedious and exhausting.Eventually, we decided to split the specifications up into the decisions that make them up, and get agents to do the boring work of tracking and bookkeeping. deccy is the first realisation of this idea. We're already working on the next iteration, but we wanted to share the mental model we developed.There's a long way to go from here, and we'd really love any feedback :) Comments URL: https://news.ycombinator.com/item?id=50483376 Points: 1 # Comments: 0

Fine-grained access control for Amazon Bedrock Managed Knowledge Base tools

Learn how to expose an Amazon Bedrock Managed Knowledge Base to AI agents as MCP tools through Amazon Bedrock AgentCore Gateway while keeping per-user access control. A request interceptor injects verified identity, Policy in AgentCore validates it independently, and a resource-based policy restricts direct access to the knowledge base.

From prototype to production: Engineering agents with Agent DLC

Agents stall between demo and release when teams lack a repeatable way to measure how good an agent is and whether the last change helped. This post covers three agent properties that break traditional CI/CD, a measured case where an aggregate metric hides a coverage gap, and Agent DLC: one lifecycle, two loops, and one explicit standard for judging every change.