data pipeline
Automated processing channel
A data pipeline is a sequence of automated processes that move and transform data from one system to another.
Raw data enters at one end, gets cleaned, filtered, or restructured along the way, and arrives at its destination ready for analysis or storage: We built a data pipeline to feed customer transactions into the analytics platform.
The metaphor evokes industrial pipelines—data flows through connected stages like oil through pipes.
- ETLSource-to-destination data process
- workflowSequential task progression
- ingestionConsuming into the body
- streamingReal-time digital delivery
- batchQuantity produced together
transform · orchestration · Kafka · scheduler · source · sink · extract · load · processing · upstream · downstream · throughput · latency · queue · connector · schema · staging · warehouse
- integrationintegration, synthesis, fusion, merging, unifying
- data systemsdatabase, schema, relational database, DBMS, SQL
- Big DataHadoop, Kafka, Zookeeper, ETL, batch processing
- data engineeringETL, batch processing, orchestration, Kafka, ingestion
- data servicesdatabases, database, cloud storage, ETL, batch processing
- streaming dataingestion, live feed, feeds, buffering, events