Answer: yesterday.
If, however, your organization lacks the funds for a time machine (along with the funds for unlimited data intelligence solutions), here are our top four picks for the situations in which you should prioritize initiating a data lineage process.
You’re planning a data system migration
Are you leaving behind Oracle for Azure Data Factory, SQL Server for Snowflake or SSRS for Power BI?
The benefits of data environment modernization are so many that if you haven’t done it yet, it’s probably on your to-do list. Gartner projects that by 2024, more than 45% of IT spending on system infrastructure, infrastructure software, application software and business process outsourcing will shift from traditional solutions to cloud. But even if you’re excited about the end result, it’s a headache and a half to get there.
The planning.
The data preparation.
The working on the old system and the new system simultaneously. (Business doesn’t stop just because you’re migrating.)
The troubleshooting when things go awry (and they often go awry).
Data lineage is even better than Extra-Strength Tylenol in making the data migration headache go away.
The planning: a key stage of data migration planning is getting an accurate picture of your current data environment and data flow – and then optimizing it. Mapping a data environment and creating visualizations of data flow is what data lineage tracking does best.
The data preparation: what data assets or processes are important to migrate – and which are outdated or redundant? With the visual map created through data lineage techniques, you can clearly see the processes that overlap and the paths that dead-end.
The working on the old and new system simultaneously: what data is where? Instead of manually floundering around, wasting time and energy trying to find the assets you need, all you need is minutes (or even seconds!) running a data lineage tool.
The troubleshooting: it’s here where data lineage analysis can really save the day (and people’s tempers). A process isn’t working the way it should? The data seems corrupted? When you use data lineage to trace any data point back to its origin, pointing out every step along its journey, you’ll be able to easily identify where things went wrong.
The data lineage process enables you to plant your feet firmly on the ground while getting your head into the clouds.
Your data team spends too much time investigating reporting errors
It’s the words that haunt your nightmares.
“Where does this number come from?”
Once you wipe the cold sweat off your forehead, the nightmare isn’t over. Because that’s precisely the question that you hear way too often from management and business users. It’s the question that foretells a postponement of all the other valuable data activities you’re working on while you comb through your data systems for the answer.
If this describes an all-too-familiar situation, then data lineage is the answer to your (bad) dreams.
When root cause analysis is a manual process, it takes forever. When you have automated data lineage at your disposal, it’s a click of a button, a wait of seconds, and a look at the resulting data lineage visualizations. Just follow the trail back until… oh! There’s the reason for that metric coming out like that!
“Where does this number come from?”, the former Question of Doom, becomes a simple way for your data team to show your mastery of your organization’s data.
You’re thinking about changing a data-related business process
Change can be a wonderful thing:
More precise processes.
More measurable metrics.
More accurate analytics.
More relevant reporting.
But often things get worse before they get better:
“What happened to my report?”
“My analytics don’t make sense!”
“Where is the data that was here yesterday?”
“This whole system is messed up!”
An ounce of prevention is worth a pounding at your door by irate business users.
Impact analysis – checking what systems, calculations, processes, reports, etc. will be affected by a proposed change – will enable you to drive progress without driving users (and yourself) crazy. But unless you have an incredibly simple and straightforward data environment (in which case, we envy you), manual impact analysis can take days, weeks or months.
When data lineage enters the picture, however, efficient and safe changes at record speed become possible. Whenever a change is proposed, data lineage automation software can start at the data elements that will be changed and trace them backwards and forwards throughout your data landscape, identifying all parts of the system that could be affected. Results are available in minutes or seconds. Insights can be had quickly and decisions can be made in record time.
We’ve seen it happen time and time again:
Two-day impact analysis processes reduced to a short one-hour routine.
Impact analysis that formerly took 100 days cut down to one day.
Change planning projects that used to take weeks now take a few minutes.
With a data lineage solution on your side, things only change for the better.
You have an audit coming up
Yearly reporting and upcoming audits spell late nights and frazzled nerves.
It’s here where a data lineage solution really shines.
When you need to prove the accuracy of your internal models or calculations for regulations like FRTB, TRIM or IFRS 17, data lineage solutions will highlight exactly where your model data is coming from, so you can easily show its veracity to auditors.
When you need to prove the existence, removal or protection of data under regulatory standards like GDPR, CCPA or HIPAA, data lineage will reveal everywhere the information in question exists in and impacts your data environment.
When you need to explain the rationale behind a number on a report, data lineage will show you exactly where the data originated and how it was transformed and manipulated to get that figure.
And best of all, automated data lineage management does any and all of this quickly and painlessly. No more late nights. No more frazzled nerves. Your data team can meet the regulatory standards while being calm, confident and well-rested.
Bring on the audits!
What are you waiting for?
Migrations, process changes, user questions and regulatory requirements are par for the course in any data-heavy organization. If you’re not dealing with one today, you’ll be dealing with it tomorrow.
And then you might regret you didn’t implement a data lineage solution yesterday.
So it’s worth starting to think about the initial steps of data lineage implementation.
After all, much better to avoid the embarrassment of trying to add “time machine” as a line item to your organization’s budget.
