Community
What is data pipeline? A data pipeline consists of a sequence of processes for processing data. The data is ingested at the beginning of the pipeline if it has not yet been loaded into the data platform. In the following phases, each process produces an output that serves as the input for the next step. This continues until the pipeline is complete. Independent steps may be executed simultaneously in certain instances. Components of data pipeline A source, a processing step or stages and a destination are the three main components of a data pipeline. Data Pipeline Considerations Numerous considerations must be made while designing data pipeline designs. For example, - Does your pipeline need the ability to handle streaming data? - What kind of data volume do you anticipate? - How much and what sort of processing does the data pipeline require? - Is the data created in the cloud or on-premises, and if so, where should it go? - Are you planning to construct the pipeline using microservices? - Is there a particular technology in which your team is already proficient at developing and maintaining?
This content is provided by an external author without editing by Finextra. It expresses the views and opinions of the author.
Oleksandr Strozhemin Co-founder and CEO at Trinetix
13 August
Elaine Mullan Head of Marketing and Business Development at Corlytics
12 August
Abhinav Paliwal CEO at PayNet Systems- A Neo Banking Software Platform
Donica Venter Marketing coordinator at Traderoot
Welcome to Finextra. We use cookies to help us to deliver our services. You may change your preferences at our Cookie Centre.
Please read our Privacy Policy.