A key knowledge pipeline functionality is to trace knowledge lineage, together with methodologies and instruments that expose knowledge’s life cycle and assist reply questions on who, when, the place, why, and the way knowledge adjustments. Information pipelines remodel knowledge, which is a part of the info lineage’s scope, and monitoring knowledge adjustments is essential in regulated industries or when human security is a consideration. Platforms which have knowledge lineage capabilities embody Alex Options, Alation, Atlan, Boomi, Collibra, Erwin, IBM, Informatica, Manta, Microsoft, Octopai, Oracle, Exactly, Secoda, Solidatus, SAP, SAS, and Talend. Different knowledge catalog, knowledge governance, and AI governance platforms might also have knowledge lineage capabilities.
“Enterprise and technical stakeholders should equally perceive how knowledge flows, transforms, and is used throughout sources with end-to-end lineage for deeper affect evaluation, improved regulatory compliance, and extra trusted analytics,” says Felix Van de Maele, CEO of Collibra.
The info ops behind knowledge pipelines
If you deploy pipelines, how have you learnt whether or not they obtain, remodel, and ship knowledge precisely? Are knowledge errors captured, and do single-record knowledge points halt the pipeline? Are the pipelines performing persistently, particularly below heavy load? Are transformations idempotent, or are they streaming duplicate data when knowledge sources have transmission errors?
