What is data lineage?
Data lineage is the data’s life cycle or the full data journey. This helps BI teams understand the data’s origin, flow, and where it exists currently. Data lineage is a visual representation of the data’s overall path. It highlights how the data is manipulated through the ETL process. Instead of sharing information between people and departments, businesses can use data lineage. This tool automatically tracks how we move data across the IT ecosystem. Additionally, if data lineage is automated, the whole process is instantaneous and the potential of error is effectively eliminated.
Data lineage shows BI teams how the data interacts with other pieces of information, is altered, and utilized in different reports. Companies will gain a deeper understanding of what happens to data as it flows through various pipelines, such as: ETL, reports, databases, files, etc. This helps keep their data environment controlled and organized.
Within data lineage, there are two main types: horizontal and vertical. Horizontal is more high level which depicts the flow of data between systems. Horizontal data lineage allows BI teams to view the entire picture but the detail only goes so far. To access a more technical view, vertical data lineage can be utilized. This view is drilled down so you can see exactly what is happening and particular transformations. BI teams should use both views in order to get the complete picture of how the data has moved within systems and between systems.
Data lineage pathways represent business processes since the processing occurs within the lineage itself. These pathways help companies develop better insights by accessing past performance. The information derived from data lineage can inform where the company should invest and any potential changes they can implement. In addition to strategy, if the company completely understands how to deal with their overall data pipelines, they can properly address data governance, data quality, and data cleansing.
Data lineage use cases
Let’s say a BI team is looking to make a change in one of their current processes. In order to understand the impact, they must be able to see how this will affect the ETL, database, and reports. Data lineage allows BI teams to view the data’s complete journey and it sheds light on how one small change can impact the entire flow. The BI team can then decide if it is worthwhile to make the adjustment and how all of their upstream processes will be affected. This helps eliminate future errors and avoid inaccuracies.
Here is another common scenario. The company wants to migrate their legacy systems to something more modern. Our BI team is back at it, looking through every table and column to make sure all of the data is successfully migrated. Without realizing it, they may be migrating tables that either lead to dead ends, are no longer in use, or are duplicated.
When it comes to pinpointing and eradicating an error, data lineage helps BI teams accomplish root cause analysis. This process allows BI teams to find the error and see where the affected field exists in all of the systems. They can see what happened along the way, how the error transformed, and which data sets it impacted. Data lineage shows a clear view of the specific error so the BI team can fix the problem, without missing a single field.
Between these projects, BI teams can spend up to 50% of their time manually searching for and understanding metadata. In addition to wasting valuable hours, data mapping is susceptible to human-error and unnecessary mistakes. The situation becomes even more complicated if someone notices a problem in a report. BI teams must backtrack and look through hundreds, if not thousands, of data sets. Even if they identify the issue, they may not have found the actual root of the error. If the root cause is not established, the business will continue utilizing incorrect reports, leading to inaccurate strategy and insights.
These situations are only the tip of the iceberg, so to speak. Without a full understanding of the data that is flowing between systems, BI teams will always be one step behind. This will negatively impact a plethora of processes, including: audits, compliance, migration, impact analysis, and root cause analysis. In order to overcome these issues, automated data lineage allows businesses to maintain complete control over their data.
