ClickHouse forecasting and anomaly detection with time series foundation models
Connect TimesFM to ClickHouse with executable UDFs to forecast retail demand, detect anomalies, and account for known future events using SQL.
Search headlines, summaries and articles to connect the dots.
Connect TimesFM to ClickHouse with executable UDFs to forecast retail demand, detect anomalies, and account for known future events using SQL.
arXiv:2611.21651v1 Announce Type: new Abstract: A controller that resamples an answer, escalates to a larger model, or abstains is usually justified by a confidence signal. Three claims about such a signal do not follow from one another: it ranks answers (discrimination), it supports withholding some of them (selective prediction), and it changes an answer for the better (correction utility). We give an exact decomposition of the third into gated recoveries minus gated harms, which bounds whether a comparison can succeed before it is run: a gate that only replaces a proposal can act only where the base policy was already wrong, so the initial error count caps the discordant pairs any correct gate can produce. When that cap falls below the count a paired test needs, the experiment cannot return a positive result for a working mechanism, nor distinguish one from a gate that never fired. We measure all three targets on two model scales and find the bound binds. In a small-model suite a perturbation-agreement verifier reaches within-policy AUROC 0.990 (95% interval [0.95, 1.00]) while every episode outcome matches single-sample decoding at 4.32 times the call count. In a controlled 840-episode study no verifier replaced the proposal on any of the 60 GSM8K decisions, so that comparison tested nothing; the entire GSM8K gain across six controllers is one task, and it is the task whose identical temperature-zero repeats disagreed, making it the one answer the service does not reproduce. On the diagnostic set the gate could reach four errors while the test needed six, so even a perfect gate had no significant result available. We report two checks that would have caught this before any budget was spent, and show that tie handling in rankings, tied selection cutoffs, and unequal output limits each move a headline number in the same dataset. The binding constraint on this literature is measurement resolution, not the choice of signal.
arXiv:2611.22847v1 Announce Type: new Abstract: As growing data volumes strain the capacity of distributed file systems, server-side storage compression transparently reduces disk usage, but its effectiveness is limited when similar blocks land in separate compression records and cannot share matches. We present a client-side reordering framework that transforms the physical block layout before persistence, so that related blocks are co-located within the same compression scope without replacing the existing storage-server compressor. It operates on bounded reordering windows, periodically selects among candidate layouts via lightweight compression of sampled data, and recovers the original logical order during reads with compact window-local metadata. We implement the framework in the SkyFS distributed file system, using the Zstandard (Zstd) compressor on the OpenZFS storage backend. Results on diverse datasets show that the framework reduces physical storage by up to 42.9% relative to logical-order writes, while achieving 94-108% of baseline sequential-read throughput and 91-99% of random-read IOPS. Client-side layout construction is the primary additional write cost, showing that adaptive similarity-aware reordering substantially improves storage efficiency with acceptable overhead.
Visualize the November 13 FAA outage using ADS-B flight data and ClickHouse’s FORMAT PNG, rendering an entire day of US air traffic directly from SQL.
In a lab near Newark Airport in New Jersey, a team of around 30 experienced human sniffers are slowly working their way through every odor on Earth. Their careful notes—“nutty,” “peachy,” “marine”—are cross-referenced with a chemical analysis to create a rich set of training data for what AI startup Osmo calls the world’s first foundation…
Local unit tests, a free dbt State plan, Wizard Desktop in Public Beta, and dbt Charts in Claude and ChatGPT.
Testing Postgres read replicas reveals why more write traffic can reduce lag, how WAL settings affect latency, and where aggressive tuning introduces tradeoffs.
Trunk cuts observability costs by up to 95%, reclaims 20 engineering hours a week, and accelerates flaky test detection with ClickHouse.
Define metrics once, give agents context, and keep pipelines fresh. See how healthcare teams build AI their care teams can trust.
arXiv:2611.14654v1 Announce Type: new Abstract: Tool-using language-model agents accumulate histories of their own mistakes, and a system that compacts, repairs or hands off such a history decides what it says about those mistakes while their effects stay in the world. Prior work cleans the history at a fixed world, where showing a corrected step and saying it is done move together. We ask whether the same correct content helps or hurts depending on how the history states it and what the world holds. We freeze an agent mid-task with its exact world state, edit the history or the world, and let a receiver model continue. On AgentCE-Bench, after the agent commits a wrong value, we hold the world fixed and add the correct value by two natural edits: a truthful note beside the preserved error or the agent's own completed action. The note raises recovery, the share of remaining targets the receiver gets right, from 51% under a matched neutral note to 66%. The completed action lowers it below the unedited history, from 50% to 44%, so the value's presence alone does not predict the effect. On AppWorld, after a database write, we vary whether its receipt reports success or failure and whether its effect is present. A success receipt lowers whole-task success from 45% to 28% while the effect is absent and raises it from 38% to 49% once present. The two effects differ by 27 points pooled over two receivers, a difference positive in six of eight receivers at 97.5% and seven at 95%. A history edit's effect thus depends on the world it describes, and a history made to look clean while the world still holds the error can leave an agent worse off than no edit at all.
Loom gives coding agents shared memory and a code index grounded in the repository, so each session can build on what the last one learned.
ClickHouse Cloud now has a Previews page where anyone can discover preview features, enable some themselves, join waitlists for others, and send feedback, with API support for automation.
From interactive maps to graph exploration, DuckDB makes it possible to analyze data without a database server. Ilya Boyandin shares how this approach led to SQLRooms, a toolkit for building analytics applications that bring data, visualizations, and AI-assisted analysis together on the user’s machine.
Our founders on the age of intelligence: agents, open data infrastructure, and the humans building the agentic future.
Connect ClickHouse to Amazon S3 Tables to query Iceberg data in place, transform retail analytics, and write reusable results back for dashboards and AI agents.
Introducing ClickHouse Observability: fully managed observability with the speed and scale of ClickHouse. Just send telemetry; we'll handle the rest. Plus, build the observability experience you've always wanted with a simple prompt in Observability Apps.
ClickHouse Managed Postgres, our NVMe-backed Postgres service natively connected to ClickHouse, is generally available, and we're acquiring Xata to add instant branching, cloning and scale-to-zero for developers and AI agents.
arXiv:2611.09953v1 Announce Type: new Abstract: Handwritten character recognition systems can achieve high accuracy while retaining concentrated errors among visually similar classes. Motivated by residual letter confusions in earlier geometry-based recognition work, we investigate whether these errors can be addressed through the training objective. We introduce Confusion-Adaptive Margin Learning (CAML), which constructs a sparse weighted confusion graph from a discovery partition held out from pilot training and applies pair-specific cosine-separation constraints during final training. CAML adds no classifier, routing rule, or inference-time parameters. We evaluate CAML on EMNIST Letters against a strictly matched cross-entropy CNN baseline, using the same architecture, final training data, deterministic augmentation, optimizer, and checkpoint selection. Eight confusion pairs are targeted: I/L, G/Q, U/V, R/V, D/O, C/E, I/J, and H/N. On the official clean test set, accuracy changes from 95.6346% to 95.6731%, and macro-F1 from 0.956326 to 0.956724. Mean accuracy across ten predefined corrupted conditions increases from approximately 95.3812% to 95.4303%; symmetric errors within targeted pairs decrease from 614 to 605 (1.47%). The global differences are small and not statistically reliable. CAML fixes 124 baseline errors but introduces 116 new errors on clean data (exact McNemar p=0.6515), and no corruption-specific difference remains significant after Holm correction. Thus, CAML does not establish global superiority, but shows that a leakage-controlled, pair-specific margin objective can alter targeted error structure with limited overall benefit for an already strong classifier.
DuckDB is fast among other reasons because it stores data by column, so a query reads only the columns it names. This post covers DuckDB features that take advantage of that design choice: querying files in place, column-level SQL, aggregation shortcuts, window functions, ASOF joins, and writing well-organized Parquet.
Autonomy is an outcome of AI maturity. Build the trusted data, governance, and context that let AI and agents act with control.