If you’re a Business Intelligence developer, chances are good it’s because you really love data and figuring out how to take a mountain of data from different sources and get meaningful, insightful, actionable information from it. You’re not in it because your family has been doing it for generations, and you may not even have a degree in it; in fact, you might not have ever heard the term “BI developer” in school. You’re in it because you dig it, and your talent for it makes you indispensable.
So why do you spend so much time not doing your job? Specifically, why do you spend so much time tracking down data errors that pop up in reports?
The BI Developer’s Lot
Yeah, sure, we get it. You do it because you’re the only one with the knowledge, skill, patience, and attention to detail. You’re the only one who knows the data environment well enough to even begin tracking down bugs and errors. You can’t assign it to the summer intern, because by the time you explain where and how to look, you could just do it yourself. But here’s the problem: As described previously in this space, there’s a big pile of manual grunt work that goes into it. It can occupy large chunks of your time. But when you go into your year-end performance review, what’s going to get you that gold star: That you tracked down and fixed 26 data errors, or that you increased company efficiency, cost control, forecasting, or whatever through new and creative ways to extract and present business-critical information?
The process of tracking down errors isn’t just laborious, it could also cause even more errors: If your company’s data environment resembles a spider web (or a bowl of spaghetti), you can easily go down a wrong path and end up making the problem worse, not better. There has to be a better way.
Automated Tools to the Rescue
It’s time to get out from under all that non-value-added activity. The path forward is automated data lineage tools. Hours/days/weeks of tedious manual work can be replaced by a few minutes of perusing a visual representation of where every piece of data in a given report came from and how it got there. This helps you answer tricky questions, such as:
– Was there a data type conversion error along the way?
– Was a data value calculated from other data fields? Is the calculation correct? Could there be a divide-by-zero somewhere?
– Were missing values properly accounted for?
– Were there other possible conversion errors (date/time formats, units of measure, time-zone corrections, foreign exchange conversions…)?
– Are different data sources speaking the same language? (Example: Do the sales territory values or customer IDs in one system match up with those in other systems?)
– …and so on, you get the idea.
An automated tool of this type can easily pay for itself by freeing you of the time-consuming, error-prone drudgery of tracking down all the possible sources of error in data’s journey from source to report. Just knowing the path narrows your search and shows you where to focus in finding the error.
Friends, it’s time to stop being the company’s data pooper scooper, and time to do the job you are actually paid to do. It’s time to stop spinning your wheels.
