Browse documentation
Understand the engine
Architecture
Follow a query through the single-process Rust engine.
A small, composable engine
Igloo is currently one Rust crate with a library and a binary, running in a single process. Apache DataFusion provides SQL planning and execution; Apache Arrow provides in-memory schemas and record batches. The connector preview registers native PostgreSQL, generic ADBC sources and named local Parquet snapshots with inferred schemas.
Native PostgreSQL uses a custom TableProvider backed by tokio-postgres. Generic ADBC sources now have a separate TableProvider using standard metadata and streamed Arrow batches. The legacy one-shot ADBC example remains separate. All routes share DataFusion; distributed scheduling is future work.
The query path
In serve mode, a PostgreSQL wire request reaches the query handler. Cacheable, zero-parameter statements first look up the normalized SQL in the result cache. A hit returns the saved Arrow batches.
On a miss, DataFusion plans against registered catalogs. PostgreSQL providers generate schema-qualified SELECT queries with supported projection and filter pushdown; the Parquet provider scans local files. DataFusion performs the remaining filtering, joins, and aggregation before collecting result batches.
Eligible results enter the cache, then the server encodes them into the client’s requested PostgreSQL text or binary representation. Parameterized statements are bound and executed without result caching.
- Client → PostgreSQL wire handler → result cache
- Cache miss → DataFusion → PostgreSQL / local Parquet
- Arrow record batches → cache (when eligible) → client
- Background JSON CDC polling → whole-cache invalidation
Source map
The code is organized into focused modules inside src/.
| Module | Responsibility |
|---|---|
main.rs / lib.rs | CLI dispatch and library entry points |
config.rs | TOML, environment overrides, validation, secret redaction |
server.rs | PostgreSQL wire protocol and query handling |
adbc_source.rs / connectors.rs | Generic ADBC metadata/streaming, connector configuration and coverage |
datafusion_engine.rs | SessionContext, table registration, query execution |
catalog.rs | PostgreSQL introspection and deterministic aliases |
postgres_table.rs | Custom TableProvider, scans, PostgreSQL-to-Arrow decoding |
pushdown.rs | Conservative expression-to-SQL translation |
cache_layer.rs | Thread-safe Arrow result cache, SQL keys, LRU, TTL |
cdc_sync.rs | JSON event discovery, path deduplication, polling |
crypto_metrics.rs | OHLCV fixtures and analytical SQL builders |
adbc_postgres.rs | Standalone experimental ADBC C driver FFI |
errors.rs | Shared error types |
Conservative filter pushdown
Supported comparisons, null checks, literal IN / NOT IN lists, and AND expressions can be translated into PostgreSQL WHERE clauses. Unsupported expressions stay local. DataFusion re-applies pushed filters because the provider reports them as Inexact.
String ordering predicates are kept local to avoid collation differences. Limit pushdown is gated when filters cannot all be translated. The differential test suite compares pushed and unpushed execution.
Arrow and Parquet today
The table called iceberg is a DataFusion ListingTable over a plain directory of Parquet files. It has a fixed user_id / data schema. It has no Apache Iceberg manifests, snapshots, transactions, or materialized views.
Actual Iceberg tables, persistent cache backends, and additional connectors are described in the roadmap.
Grounded in the engine source at connector preview b5a210d (unreleased) . The quickstart pins this experimental source revision; it is not a tagged release.
Suggest an improvement