On a scale of 1 to 10, rate the difficulty of the following tasks:
- Searching for a needle in a haystack
- Finding an earring that fell off in the Mall of America
- Tracking down every appearance of a given customer’s birthdate amongst 100K+ data assets across your entire BI landscape
Haystacks and gigantic malls have NOTHING on data repositories.
Use it or lose it
Unlike the rock collection or shell collection you may have had as a child, you don’t collect data in order to have a data collection. You collect data to use it.
Data needs to be accompanied by the metadata that explains and gives it context. Without metadata, data is just a bunch of meaningless, unspecified numbers or words that are about as useful as a bunch of rocks (or shells).
And without effective metadata discovery capabilities, metadata isn’t all that useful either. What good is detailed metadata if you don’t have an efficient way of querying and searching it?
Metadata discovery is what turns your metadata into beacons of light that point the way to the data you need. It’s what turns data collection into data empowerment.
Let’s take a look at several ways that metadata discovery unleashes the power of your data.
Finding data for regulatory compliance like GDPR
“Regulatory compliance” is a delicate way of expressing the reality:
Powerful authorities regulate.
You comply.
Or else.
True, we can often see the value behind many of the regulations (I don’t want my private data misused any more than the next guy), but still: compliance is a hassle and a half!
“Delete my personal information! All of it! On the double, pronto and ASAP!” demands Sonya from Spain. The specter of GDPR rears its ugly head, and your BI team burns the midnight oil combing through table after table to find every instance of Sonya’s name and birthdate (and of course different BI systems use different labels for the “birthdate” and even the “first name. It just would be that way, wouldn’t it?)
In cases like these, metadata discovery saves immeasurable time and patience. You would be able to locate private and sensitive data quickly, deal with it, and prove you dealt with it. If a BI metadata management automation tool is designed with forethought, sensitive data can be tagged from the outset, making the discovery process even smoother.
Identifying tables or fields that you need to change
Nothing makes hindsight work even better than 20/20 than the threat of serious inconvenience.
When they started using zip codes in 1963, they should have realized they would eventually want to be more accurate and add more numbers! Now we have to identify all our zip code fields and change them from five to nine digits!
When they wrote computer programs in the 1960s, they should have realized that using only two digits to signify the year had the potential to cause havoc when we reached the year 2000! How are we going to change the year format everywhere in our systems?!
Unfortunately, all these shortsighted programmers and postal workers aren’t around to defend themselves against your cry of desperation at needing to make a change to a field or other data asset EVERYWHERE in your BI environment.
Implementing such changes manually – searching and searching to find every place where the field in question is located, taking into account that different BI systems may use different names for the same field, plus working to understand the impact analysis of making the necessary change – is enough to drive a BI team over the edge.
Intelligent metadata discovery saves the sanity of your BI team. It’s simple to search for the data asset in question and pinpoint every single place it appears in your BI landscape. Even major changes become only a minor inconvenience.
Finding the data you need to transfer during migrations
When birds migrate, they take nothing but the feathers on their backs. When BI systems migrate, it’s another story.
- What data will you take with you to the new system?
- What data will you leave behind? (Many systems charge on a usage or storage basis, making optimizing what you transfer a move that not only saves migration time but makes good financial sense.)
- Which assets in the old system map to which assets in the new system?
BI and IT migrations can take months (if not longer), much of which is just trying to understand what should come with you!
Automated metadata discovery empowers you with reduced time, effort and error in preparing for BI system migration.
You’ll feel free as a (migrating) bird!
Powering automated data lineage
Data lineage gives you the full story of your data: where it came from, where it went, and what happened to it along the way. Automated data lineage enables quick, efficient root cause analysis, impact analysis and other mission-critical BI capabilities.
Automated data lineage is built off of automated metadata discovery. The metadata discovery tool has to identify the dots of metadata before the data lineage tool can play Connect the Dots.
Powering data catalog automated creation and updates
A data catalog is a tool that organizes all the data assets in a company’s data landscape, for each one including definitions, descriptions, ratings, responsible individuals, and more.
A data catalog creates company-wide consistency around your BI assets. It provides your data citizens with independence in understanding, locating and using data they can trust.
Where do all those data catalog features like definitions, descriptions, etc. come from?
Most of it is in the asset metadata, of course.
But try harvesting all the metadata by hand, and by the time your catalog is ready, it’ll be time for a new edition.
Enter automated data catalog creation and updates, powered by – what else? – automated metadata discovery. Metadata discovery tools harvest the asset metadata so it can be processed by the data catalog solution during the initial catalog creation. They also periodically recheck the metadata of all the data assets throughout your BI landscape – and update your data catalog metadata accordingly.
Switch on the Discovery Channel
In a world of data, data and more data (not to mention data), metadata discovery is your ticket to harnessing your data and making it work for you.
And now that you can dip into your massive amounts of data and pull out the exact data asset you need, it’s time to move on to other challenges.
Like finding your car keys.
