Concurrency

Pipeline

Chains processing stages connected by channels.

Data transformationStream processingETL jobs

Understanding Pipeline

The Pipeline pattern chains multiple processing stages together, where each stage is a goroutine connected to the next by a channel. Data flows through the pipeline like water through pipes — each stage receives input, transforms it, and sends the result to the next stage. In Go, each stage is a function that takes an input channel and returns an output channel, making stages composable. This pattern is Go\'s idiomatic approach to stream processing, ETL jobs, and any data transformation that can be broken into sequential steps.

Key Concepts

  • •Stage — a function that reads from an input channel, processes data, and writes to an output channel
  • •Channel linking — the output channel of one stage becomes the input channel of the next
  • •Concurrent stages — all stages run as separate goroutines simultaneously, processing different items
  • •Backpressure — if a stage is slow, its input channel fills up, naturally slowing down upstream stages

When to Use

✅ Use when
  • • Data processing involves multiple sequential transformation steps
  • • You want concurrent execution of different pipeline stages
  • • You\'re building ETL, stream processing, or data transformation workflows
  • • Each processing step is independent and can be tested/reused separately
⚠️ Avoid when
  • • The transformation is simple enough for a single loop
  • • Stages have complex error handling that\'s hard to propagate through channels
  • • The overhead of goroutines and channels exceeds the benefit for small datasets

Structure

Source (generator)
↓ chan
Stage 1 → chan → Stage 2 → chan → Stage 3
↓ chan
Sink (consumer)

How It Works

Read
Parse
Enrich
Format
Stages connected by channels
1

Pipeline Stages

Define processing stages connected by channels.

1 / 4

Basic Implementation

Number transformation pipeline:

main.go
Loading editor...

Real-World Example: Log Processing

Multi-stage log processing pipeline:

main.go
Loading editor...