David, the Chief Business Analyst for a medium-sized software company, walked into the office on Monday morning and greeted his staff with a smile and encouraging words. They had great numbers to round off Q3 and David was looking forward to sharing that exciting information at the shareholders meeting that afternoon.
But as he collected the Q3 report that he had just sent to the printer, a considerable inconsistency glared up at him.
Not today! David thought. This error required fixing, which meant that he and his team would have to trace the data’s journey back to its source.
He quickly rounded up his team of analysts and they immediately began mapping the data lineage of the specific data point in order to determine where the inconsistency was made. David hoped that other data hadn’t been botched as well.
The Head of BI knew that this process would take hours if not days, so he postponed the shareholders’ meeting, much to David’s dismay.
Why does this process take so darn long?
Before automated data tools were introduced, whenever an inaccuracy was identified in a data set, BI teams had to manually trace the data or manually map out the entire enterprise data lineage. The goal was to locate, discover and fully understand all of the alterations that the particular data point went through before it landed in a specific report. With data scattered across multiple systems, this was an error-prone, very time-consuming and not to mention, costly process.
Once the business data lineage is mapped out for them, the BI team can then analyze which database tables and ETL (extract, transform, and load) processes were both involved in and affected by the specific inaccuracy. This involves going through each dataset and checking if it was altered by this error. After completing this extremely timely task, David will then receive his updated reports, which may be days or even weeks later than originally expected.
Automated Business Data Lineage
In order to save time, energy and the inevitable headache, BI teams can utilize visual mapping, such as an automated data lineage tool. This system tracks the entire journey of the data and illustrates its life cycle. It follows the data as it flows through multiple systems and notes any changes that it has encountered.
Data lineage mapping tools, such as Octopai, work automatically to easily and quickly map out data’s entire lineage. Business data lineage looks like a detailed visual map that shows BI teams where each individual calculation was made in every system leading up to the current report. On one screen this tool shows all objects – such as processes and tables – related to the calculation throughout the entire landscape.
The Benefits of Data Lineage Mapping
Data Lineage provides Business Intelligence professionals with easily accessible information, so they can implement changes accurately, in a fraction of the time. This especially comes in handy when BI teams plan to run an impact analysis, allowing them to find and isolate errors instantly. Business data lineage also assists companies when they need to migrate their systems and move data over from one server to another.
With accurate enterprise data lineage in place, data is kept intact and its entire history is accessible. This is a major key to ensure a secure data governance strategy. In terms of data lineage mapping for businesses, this is especially important for GDPR, CCPA, and BCBS 239 compliance.
Data Lineage In Action
Now, when David encounters an error in his reports, he or another Business Analyst can quickly locate where and why the mistake was made. They can then easily correct it by utilizing Octopai’s automated data lineage system. Within moments, David can view the entire journey that the data took before it ended up in his report. This allows him to spot the mistake, maintaining the accuracy of his entire report. With automated business data lineage in his toolbox, David feels a renewed sense of confidence ahead of Q4. Octopai’s automated data mapping tool enables the BI group to not only locate data instantly, but to understand everything about their data – ranging from where it originated, to any changes it underwent, all the way to where it is headed.
Updated October 2020
