Current work / Data infrastructure Pre-alpha

OpenETL

Data movement.
Your infrastructure.

I’m working on OpenETL, an open-source platform that moves tables, changes and events from where they’re made to where they’re used. Run it yourself, define pipelines in YAML, and choose scheduled syncs or continuous streaming.

OpenETL provides the platform. Users can build connectors for any source or destination they need, alongside the official connectors. More official connectors are coming soon.

Focus: Extensible data movement platform
Stack: Go, Rust, Arrow, React
License: Apache 2.0
Stage: Pre-alpha · APIs may change

01 / The problem

Moving data is only half the job.

Getting rows from a database to a lake is the visible part. The harder questions show up when a process crashes, a schema changes, or a source sends a record again.

OpenETL brings those concerns into the pipeline: checkpoints to resume from, policies for schema changes, and delivery guarantees evaluated for each stream. The CLI and console make that work inspectable.

02 / The architecture

Control the work.
Keep the data flowing.

Define & observeCLI + Console

Go control planeAPI + OrchestrationPostgres metadata

Dispatch work & coordinate runners

Source examplesPostgres · Kafka · Kinesis+ your own connectors

Rust engineNormalize & routeApache Arrow batches

Destination examplesIceberg · S3 / ParquetPostgres · Kafka · Kinesis+ your own connectors

Simplified architecture with example connectors. Extend either end with your own sources and destinations. Runners host connectors, drive the engine, and coordinate checkpoints in object storage.

Go for coordination

The API, orchestrators and runners manage pipeline execution. The CLI and React console share the public API.

Rust for the data path

The engine normalizes records into Arrow batches, handles schema policies, and routes data to writer lanes.

03 / Engineering details

Make failure part of the design.

Bring your own connectors

Build source and destination connectors for the systems you need. OpenETL provides the connector protocol and a Go SDK; external connectors can be written in any language. HTTP API sources can also be defined declaratively in YAML. Read the connector development guide. More official connectors are coming soon.

Delivery guarantees, per stream

The planner evaluates source replay and destination capabilities. A pipeline is rejected if its computed delivery level falls below what it requires; exactly-once delivery depends on that combination.

Checkpoints with a commit point

Checkpoint records live in object storage. Conditional updates and fencing stop older attempts from committing over a newer runner, while recovery uses the recorded destination state.

Configuration you can review

Pipelines live in YAML. Plan changes before applying them, then use the console to inspect freshness and failures. Metrics and traces integrate through OpenTelemetry.

04 / In practice

A pipeline lives in git.

This excerpt schedules generated rows into a local Postgres destination. Source and destination resources are declared separately in the complete example.

apiVersion: openetl.io/v1alpha1
kind: Pipeline
metadata:
  name: faker-to-postgres
spec:
  schedule: { every: 1h }
  source: { ref: faker }
  destination:
    ref: local-postgres
    namespace: faker

From the repository’s faker-to-postgres example. OpenETL is evolving; the source has the latest format.