apache/iceberg

Apache Iceberg is worth adopting when multiple engines or teams need to share large analytic tables rather than one warehouse owning them. This repository is the Java reference implementation; other language implementations live in separate projects.

The problem

A team has petabytes of Parquet or ORC files in object storage and wants to query them from Spark, Trino, Flink and Hive. Without a table format, the engine sees a folder of files: file lists are stale, there are no real schema changes, and concurrent writers create partial or conflicting results. With Iceberg, the engine reads a table instead of a directory, so the same data is queried consistently and writers coordinate through table metadata. The result is a single table that several engines can safely work with at the same time.

What's different

Iceberg separates table metadata from the data files. The format specification defines snapshots, manifests and a metadata layer, and the Java library here is the reference implementation that other libraries follow. In practice, a processing engine depends on the core module, and Parquet, ORC and Arrow support are optional modules. The README organizes the code around exactly this split: iceberg-core is what engines should depend on, while iceberg-parquet, iceberg-orc and iceberg-arrow are optional.

The practical benefit is engine choice: the project states that Spark, Trino, Flink, Presto, Hive and Impala can work with the same tables simultaneously, and engine integrations are shipped as separate modules. The trade-off is that Iceberg is infrastructure, not a standalone database. You do not run it directly; you choose a catalog and a compute engine, then add the engine integration. Compatibility therefore depends on the versions of Spark, Flink or Hive you run, which the project tracks on its multi-engine support page. Runtime jars are available for Spark and Flink to avoid dependency conflicts, but that packaging choice is also yours to make.

Who should try it

A good fit is a data platform team that already stores large tables in object storage and needs more than one engine to read them, or needs a JDBC-style table abstraction across Spark and Flink jobs. A poor fit is a developer who just wants to query a few local CSV files, or a small single-engine warehouse where the existing table abstraction is enough; adding Iceberg introduces catalog, version and integration decisions that do not pay off at that scale.

This repository is Java-only. If your stack is Python, Go, Rust or C++, the README points to separate implementations, so check that project before committing your pipeline to this one.

Requirements and limitations

Building the Java library requires Gradle with Java 17 or 21, and running the tests requires Docker. That matters if your CI environment cannot run containers or your build image does not provide those JDK versions. Using a released version does not require a build, but engine compatibility still has to be checked against the multi-engine support documentation before upgrading.

Ongoing upkeep is mostly version alignment: each engine integration has a submodule per Spark version, and the project continuously adds features to the format. The core library is the reference implementation, so the other language implementations may not match it exactly. How far each one has caught up is not stated in the supplied material and is worth checking if you rely on one of them.

HotRepo's take

HotRepo recorded one daily-board observation: on 2026-10-28 the project held 9,379 stars with a gain of 6 that day. That is a single dated snapshot, so it says nothing about whether interest is rising or fading. An eleven-year-old Apache project does not need a trend argument anyway. The real decision is whether your engine versions match what Iceberg currently supports and whether you are prepared to own the catalog and integration choices. Verify that against the multi-engine support documentation before upgrading.