Where did the data in this report come from? You may have heard this question time and time again, no matter the vertical. Whether it be banking, healthcare, education, or SaaS, companies constantly wonder about the validity of their data. So, how can BI and analytics teams truly prove that the data can be trusted? Although what they say may be important, what they show makes even more of an impact. Automated data lineage allows BI and analytics professionals to visually present where the data came from, what happened to it along the way, and where it exists currently. Without this concrete evidence, BI team insights derived from the data may not be regarded as highly or implemented fully.
The webinar, “Why Automated Data Lineage is a “Must Have” for BI and Analytics,” featuring Philip Russom from TDWI, Mark Horseman from Northern Alberta Institute of Technology (NAIT), and Octopai CEO Amnon Drori, focused on this issue. They looked at the importance of automated data lineage and how crucial it is for BI teams. With the many challenges and obstacles that BI and analytics professionals face, it allows them to fully ‘drill’ down into the data. BI teams can support their findings and this provides a basis for their suggestions and strategies. Automated data lineage also enables faster, easier system migrations, upgrades, expedited impact analysis, and regulations compliance.
Increase Trust and Reliance in Data
When looking at proposed strategy or insights, business owners may wonder about the validity of the data. Some common questions posed to BI teams include: Who has used this data? What is the data’s quality? How trustworthy is this data? How did you aggregate and transform it? Instead of scrambling to provide answers to each individual query, BI and Analytics teams can rely on automated data lineage.
This powerful tool provides accurate information about the data so the numbers can be trusted and the company as a whole will feel confident about future decisions. Mark Horseman from NAIT pointed out that, with automated data lineage in place, they see an increase in data usage outside of the BI group. It allows BI professionals to pinpoint exactly where they found the data by illustrating the data’s entire journey. This ranges from point of origin to where it is located today. Without fully explaining the data’s origins and transformations, it is much harder to prove the data’s credibility. By seeing the data with their own eyes, other teams will believe in the numbers. As found in the poll taken during the webinar, 60% of participants believe that enhancing the capabilities of the BI team increases business users’ trust in their data. Additionally, a minimal 6% of participants feel extremely confident when a business user asks where a particular data item came from.
Visual Representation of Data
According to Philip Russom, “a data lineage map is an inventory of data to be queried, browsed, searched.” With an illustrated map, BI teams can identify old and new patterns. This is extremely important if the company decides to perform any upgrades or migrations. A visual representation of the data allows BI teams to accomplish easier source to target mapping and can account for any possible roadblocks, dead ends, or transformations. The nature of a visual data lineage data map allows BI professionals to scan data automatically. They can gain a better understanding of the bigger architectural view, dependencies, and complex data environments. With this essential information in hand, they can address many pressing issues that businesses may face.
NAIT utilized a data lineage map while undergoing a large integration. With data streaming from all different databases, there were many steps involved and data sources at play. By implementing automated data lineage, they could see exactly where the data was going out and where it was coming from. As Mark Horseman reflected on the process, “now we have an answer at our fingertips.”
Smoother Migrations, Impact Analysis and Compliance
With automated data lineage, BI teams can see the full picture. This is vital when it comes to migrations and impact analysis. As data may be scattered between different vendors and systems, automated data lineage leverages the power of the cloud. You don’t need individual, “silo-ed” tools to accomplish a clean migration or impact analysis, you can rely on automated data lineage. In order to get your data into the best shape possible, the BI team will need to completely understand which data sets are utilized and which can be archived or removed. Once the data is cleaned up, your BI team can see how the data flows and where data exists in each system. This allows for a smoother and quicker migration or impact analysis. As found in the webinar poll, a shocking 52% of participants said it would take a few weeks to a few months to conduct impact analysis to make a change to a field in the ETL or database.
In terms of compliance, as stated by Philip Russom, “data lineage is the proof.” Whether it be an audit, CCPA, GDPR, or BCBS 239, it provides accurate, correct, and the most up-to-date information. When NAIT needed daily COVID-19 reports completed by students, they turned to automated data lineage. By filling out a form that states “I don’t have symptoms, I can come to school,” they can keep track of who is in school and this is essential for both compliance and safety. As such, these forms need to be filtered daily and easily accessible if any checks or health issues should arise.
The Future of BI and Data Lineage
It is promising to see that close to 73% of participants polled believe that automated data lineage is taking their BI operation to the next level.
As pointed out in the webinar, automated data lineage is so clear and straightforward, it is possible to even perform reverse engineering. This means that another team member can easily understand a BI product built by someone else and replicate it. Data lineage allows for self-service and provides a foolproof way to maintain compliance, track PII, perform migrations, and successfully promote data-driven development. This solution is truly the drill that can help companies unearth precious data.
