Find your next open-source project

Search by name, purpose, or keyword — or paste a GitHub repository URL

datazip-inc/olake-go

Data

Data ingestion engine syncing databases, Kafka and S3 into Apache Iceberg or Parquet for lakehouse pipelines.

2,013
311
Go
Apache-2.0
2 years old

What it does

OLake Go is an open-source ingestion engine that copies data from operational sources into open lakehouse formats. It reads from PostgreSQL, MySQL, MongoDB, Oracle, DB2 and MSSQL, plus Kafka topics and S3/S3-compatible storage, and writes Apache Iceberg tables or plain Parquet files. Each connector supports full loads and, where listed, incremental sync or change data capture.

On the source side, CDC reads PostgreSQL pgoutput, MySQL binlogs and the MongoDB oplog; Oracle CDC is marked work in progress and DB2 has no CDC entry. On the destination side, Iceberg can use Glue, Hive, JDBC or REST catalogs (Nessie, Polaris, Unity, Lakekeeper, S3 Tables), while Parquet writes to a filesystem. The project positions itself as avoiding Spark, Flink, Kafka Connect and Debezium in the ingestion path, with a self-hosted UI and a CLI for automation.

Quick start

The recommended path is the UI stack via Docker Compose. Docker and the Compose plugin are required; the command below pulls the compose file from the separate olake-ui repository and starts the stack:

curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-compose.yml | docker compose -f - up -d

Then open http://localhost:8000 and log in with the default admin / password credentials. From the UI, create a source connection, create a destination, then create a job: name it, set a schedule, pick source and destination, choose the streams (tables) and select CDC, Full Refresh or Incremental before running it. The first-pipeline walkthrough covers the same flow in detail.

CLI-oriented deployments are also documented for a standalone Docker container and Kubernetes via Helm. The CLI exposes spec, check, discover, sync and clear-destination commands; configuration and command examples live in the docs rather than the README.

Use cases

Typical uses are moving OLTP tables into Iceberg without a Spark/Flink stack, running BI tools such as Trino, Athena or Snowflake on fresh CDC data, and keeping one copy of data for both analytics and machine learning. Ready-made playground guides exist for Iceberg with Presto, Trino, Athena and Snowflake.

Constraints and caveats

  • CDC availability differs by source: Oracle CDC is unfinished and DB2 lists no CDC; Kafka sync is a bounded incremental read up to a latest offset, not continuous streaming.
  • The Quickstart compose file comes from the olake-ui repository and must be reachable from the host running Docker.
  • The default login credentials are public, so change them before exposing the UI beyond localhost.
  • Table maintenance is not part of this repository; it lives in the companion OLake Fusion project.
  • Performance figures such as 580K RPS for Postgres are the project's own benchmark results as of the stated dates and depend on their benchmark setup; they are not independent measurements.
  • Licensed under Apache-2.0.

HotRepo's take

On the Go daily list for 19 November 2026 OLake Go recorded 2,007 stars and 310 forks, with 115 stars gained that day and a best position of No. 1 on that list. The ranking is a single daily snapshot in the Go list, not an all-language record, so it shows a strong one-day showing among Go projects rather than a sustained trend. The project's technical bet — Iceberg-native ingestion without Spark, Flink or Debezium, with several enterprise database sources — is the part worth watching, especially as Oracle CDC and Iceberg V3 support are still on the roadmap.

Written by AI from the project's README, updated . The project's own docs are the source of truth.

You might also like

  1. debezium/debeziumOpen-source change data capture platform that streams database row-level changes to Kafka or directly into applications.Java13,305
  2. apache/icebergA table format that lets several query engines read and write the same huge analytic tables concurrently.Java9,379
  3. airbytehq/airbyteOpen-source ELT platform that syncs data from APIs, databases and files into warehouses, lakes and AI applications.Python22,325
  4. apache/seatunnelMove large volumes of data between systems in batch, real time or CDC, using pluggable connectors and a choice of execution engines.Java9,776
  5. apache/kafkaA distributed event streaming platform for moving and retaining high-volume event data between services.Java34,115