Databricks LTAP Revisited: The Compute Cost of Bridging Two Engines

A few weeks ago we published an analysis of Databricks LTAP and the architectural questions it left open. One question in particular stood out: if a transaction commits in Postgres but has not yet landed in object storage, can an analytical query trust what it reads?

Since then Databricks published a detailed engineering post on the architecture, and independent engineers have done their own analysis of how LTAP works. It seems it can can handle the freshness problem, however the mechanism it uses carries a performance cost that was not visible in the announcement, and it has direct implications for anyone building agent systems on top of it.

How LTAP Actually Achieves Freshness: The Union Read

To understand the cost of LTAP, you first need to understand the data path.

When a transaction commits in Lakebase (Postgres), data travels a multi-hop path before it is visible to the analytical engine. The compute node writes WAL records to a Paxos-style safekeeper quorum for durability. A pageserver materializes pages on demand from those WAL records. Background compaction threads then convert row-format data into Parquet files on object storage. Each hop introduces latency. Data that is committed and durable in Postgres is not immediately present in the columnar store.

To bridge that gap, LTAP uses what engineers analyzing the architecture have called a union read. When an analytical query executes, the engine simultaneously reads from materialized Parquet files on object storage, in-memory Apache Arrow buffers holding un-flushed transactional state, and any pending updates and deletions not yet compacted into the columnar store. It merges, sorts, and deduplicates these at query time to produce a consistent result.

This solves the freshness problem we raised in the original post. But it means the query engine is performing data transformation work at runtime that a unified system would never need to do. The pipeline between the transactional and analytical worlds did not disappear. It moved into the query path.

What the Union Read Costs

The size of the Apache Arrow buffer a query must merge against grows with write throughput. Under sustained agent workloads, that buffer is not small. Query latency is not a stable characteristic of the system. It scales with write load in a way that Databricks has not yet published data to bound this.

The union read also reconciles two fundamentally different data structures at runtime: columnar Parquet files optimized for sequential scan, and row-oriented Arrow buffers optimized for transactional writes. That reconciliation consumes CPU on every analytical query. Background compaction threads compound this, running continuously alongside live workloads rather than truly in the background. Additionally, the translation between formats is not cheap either.

Databricks' published Lakehouse//RT benchmark is TPC-H Q6: a single-table scan with no joins and no concurrent write load. It does not reflect the union read cost under production conditions. Until Databricks publishes performance data under concurrent write load, the latency profile of analytical queries in production is as an open question.

The Third Path Databricks Did Not Take

Reynold Xin, Chief Architect at Databricks, framed the history of this problem as a binary choice: HTAP tried to unify transactional and analytical workloads through a single engine and failed. LTAP unifies them through a single storage layer. However, the union read is the cost of that choice.

But that framing assumes the engine itself cannot be rethought. It assumes that MVCC, slotted pages, WAL double-writes, and process-per-connection concurrency are fixed constraints to work around rather than architectural decisions that can be unmade.

The reason previous attempts at HTAP failed is because every attempt started with an existing construct and tried to extend it to handle analytical workloads. The contention was structural because the engine was never designed to avoid it. Extending a system built for one workload to handle another does not change what it is at the core.

RegattaDB starts from a different constraint. Rather than extending an existing engine or bridging two engines at the storage layer, it rebuilds the engine from the ground up with the requirement that all three workloads, transactional, analytical, and vector, run simultaneously without contention.

A few specifics on what that means in practice:

• Version management. 
  RegattaDB replaces MVCC's VACUUM-based cleanup with the Ranger, a patented technology that tracks exactly which row versions became unneeded at the moment a snapshot is removed. There is no crawling, no approximation, and no sensitivity to the oldest active snapshot. Long-running analytical queries do not inflate transactional version debt because the version management system was designed from the start to handle both workloads concurrently without penalizing either.
• Storage layout. 
  RegattaDB uses a log-structured storage layout rather than slotted pages. This eliminates the read amplification that Postgres's page layout produces under mixed workloads, avoids TOAST indirection for variable-size rows, and removes the need for runtime reconciliation between row and columnar formats. Scans read only the data that is actually needed.
• Writepath. 
  RegattaDB writes the row data directly to its log-structured layout without a write-ahead log (WAL). This removes the double-write I/O penalty that makes the Postgres write path expensive under concurrent load and eliminates the WAL as a bottleneck in the path between a committed transaction and a queryable result.

The result is not a bridge between two engines with a union read at the crossing point. It is one engine, built from the storage layer through the concurrency model and query parser, so that the conditions requiring a bridge never arise. There is no Arrow buffer to merge against because there is no separation between the transactional and analytical execution paths to begin with.

What This Means for Agent Workloads

For agent systems the union read has a specific consequence. An agent decision loop that triggers an analytical query under high write load is not getting a clean columnar scan at predictable latency. It is getting a union read whose cost scales with the size of the un-flushed Arrow buffer, which scales with write throughput. For agents where decision latency matters and write load is sustained, that variability is a structural property of the architecture, not a configuration problem.

Vector search compounds this. pgvector runs directly on the Postgres process, competing with the same write path that feeds the Arrow buffer. Under concurrent transactional load, vector index maintenance degrades query latency for both workloads at the same time. All three workloads, transactional, analytical, and vector, are contending on the same compute with no architectural separation between them.

RegattaDB runs all three workloads under a single concurrency model with no union read, no cross-engine coordination, and no runtime format reconciliation. You get all three with great performance. The latency profile of an analytical query does not change as write throughput increases because the architecture does not require a merge step between two different representations of the same data.

Ultimately, with RegattaDB, you can scale both (guaranteed consistent) transactional and (completely currrent) analytical workloads and have them run within the same engine as you execute vector and semantic search. You leverage the aggregate compute resources of the cluster for distributed and parallel computation of these demanding queries.

What We Are Watching

LTAP is not yet generally available. Lakehouse//RT is in beta. The union read mechanism described here is based on published architecture documentation and independent engineering analysis, not production benchmarks. Databricks may publish performance characterization of the union read under concurrent write load, and if they do we will update our analysis accordingly.

What is clear from the architecture is this: the union read is an elegant solution to a problem that a different architecture would never have. Databricks correctly identified that the fragmented data stack is the bottleneck for AI agents. LTAP is a meaningful step toward solving that problem for organizations already in the Databricks ecosystem. But bridging two engines at the storage layer, however cleanly, is not the same as building one engine that was designed to run all three workloads from the start.

For architects making data infrastructure decisions for agent systems, that distinction is worth understanding before the design is set.

NOTE: Some of the references we used to gain further insight into the architecture of LTAP are the following:

• Databricks engineering post: https://www.databricks.com/blog/lakebase-ltap-rethinking-database-storage
• Amit Singh Rathore analysis: https://asrathore08.medium.com/ltap-is-databricks-finally-delivering-the-promise-of-htap-b18e74624b26
• The Build pg_lake vs Lakebase: https://thebuild.com/blog/pglake-vs-lakebase-two-very-different-things-called-postgres-lakehouse/