The life of BI (business intelligence) developers is one of uncertainty; they wake up in the morning unsure of what each day will bring. But this isn’t a starry-eyed sense of wonderment about what new opportunities will come. Far from it. It’s more akin to a nagging sense that, soon—and there’s no telling when—a new data error will surface that will require immediate attention.
The only constant is change
It’s tough to be a BI developer. We should know—our company was founded by two of them. We’ve been in the trenches and we’re well-acquainted with how pervasive this uncertainty is. The BI industry is still young, and it’s growing fast. But with fast growth comes growing pains that, understandably, keep BI developers up at night – and sometimes all night.
BI developers start their day with a set schedule based on the various projects they have lined up. They know which clients have requests in the queue and what will be expected of them. But minutes after arriving to the office – sometimes even before they’ve had the chance to grab a cup of coffee – they get that dreaded phone call. Ughhhh. That call that says there’s an error in a client report, which needs immediate attention. When this happens, the BI developer’s established schedule goes out the window, and they’re now on damage control until further notice. Joy.
On the hunt
With any luck, the BI developer is able to locate the origin of the error and sort out the problem in a matter of hours, but this doesn’t happen very often. Usually it takes days or even weeks to locate the source of a reporting error because it requires them to manually search through the data throughout the various systems, which takes forever and is extremely inaccurate.
BI developers are responsible for managing a company’s BI data across each of its multiple systems. They build ETL processes that pull data from source systems and use them to generate specific reports. So, when an error pops up, BI developers apply this process in an effort to figure out the root cause of said error. They begin by analyzing data—every process, every platform, and every data item from the client’s systems. This usually means they have to communicate directly with the client and other team members in order to fully understand the exact problem, and from there, they look at existing processes and whether they need to create new ones to get the job done.
If the client’s request is based on a source system process that already exists, then the data doesn’t need a new connector, only modifications. But, if the process doesn’t already exist then they need to create a new process from scratch. Either way it’s tons of work. And as long as they’re working on getting to the bottom of this bug, all other work is put on hold. A total standstill.
Knowing where to look is more than half the battle: data lineage automation
BI developers are wasting a ton of time searching through all the systems because they often don’t know where to look for what they need. Tools that provide BI groups with automatic data lineage visualization are speeding up this process, by showing a very clear map of the data flow, from where it originated, how it was transformed, which calculations it’s part of etc. Automated data lineage enables the BI developer to quickly locate where and why the error occurred, and easily correct it.
Redefining your BI development strategy
The BI workflow can depend on one true thing: BI developers, managers, and analysts never really know what’s in store for them each day. It might seem like the nature of the beast in the BI world—but it doesn’t need to be. New systems of automated metadata management have entered the market that completely redefines metadata discovery. We’re talking systems that can trace data lineages, automate table generation, and provide better intelligence at every point of decision-making. In short, it’s time to start looking at solutions that add efficiency—not effort—into your BI strategy.
