Changelog

v2.4.0 Latest

January 20, 2026

Added GraphQL Subscriptions source connector via graphql-ws protocol
New Cast transform for type coercion (string to datetime, int, float, bool)
Pipeline metrics now include per-stage P99 latency histograms
Kafka connector upgraded to librdkafka 2.6.0 with cooperative sticky rebalancing
Fixed a race condition in multi-worker shutdown that could cause records to be processed twice

November 8, 2025

Fixed PostgreSQL CDC source losing WAL position after connection retry
Fixed Avro deserializer not caching schemas from Confluent Schema Registry
Reduced memory usage of ParquetSource by streaming row groups instead of loading entire files
Bumped minimum Python version to 3.10 (dropped 3.9 support)

September 12, 2025

Added ClickHouse native protocol connector with async insert support
New Aggregate transform for tumbling and sliding window aggregations
Server-Sent Events (SSE) source with automatic reconnection and last-event-id tracking
YAML config now supports ${VAR:-default} syntax for optional environment variables
Improved backpressure propagation latency from ~200ms to ~5ms via lock-free ring buffers

June 3, 2025

Added Azure Service Bus and Azure Blob Storage connectors
New Enrich transform with GeoIP lookup and user-agent parsing
Dead-letter queue support: route permanently failed records to S3, Kafka, or local files
CLI: pipelinecast validate command to check config without starting the pipeline
Fixed S3 sink not closing multipart uploads on pipeline shutdown

March 17, 2025

Added Google Cloud Pub/Sub and GCS connectors
New WebhookSource with HMAC signature verification (SHA-256, SHA-1)
Prometheus-compatible metrics endpoint at /metrics
Buffer overflow policy: choose between block, drop_oldest, and drop_newest
40% throughput improvement in JSON serialization via optional orjson integration

December 9, 2024

Major release: redesigned pipeline API with fluent builder pattern
Breaking: Pipeline.add_stage() replaced by .source() / .transform() / .sink() chain
Breaking: stage lifecycle changed from start()/stop() to open()/close()
Multi-worker execution with automatic partitioning and thread-safe buffers
YAML configuration file support with pipelinecast run --config
22 connectors at launch: Kafka, RabbitMQ, Redis, PostgreSQL, MySQL, MongoDB, S3, local files, JSON, CSV, Avro, Parquet, and HTTP
MIT License

March 2024 — November 2024

The 1.x series used an imperative add_stage() API. It is no longer maintained. See the migration guide for upgrading to 2.x.