“Garbage in, garbage out.” – George Fuechsel, early IBM programmer
GIGO may be an old refrain, but it likely gave rise to the concept of what we know as ‘business intelligence’. BI has helped enhance a company’s ability to process large amounts of data. Although this may be true, analysis loses its usefulness the longer it takes to generate. After all, how do you adjust this month’s operations based on last month’s data if it takes two weeks to finally receive the information you need?
This is exactly how Octopai customer, Farm Credit Services of America (FCSA), felt when their BI team needed to modernize their data warehouse. Based in Omaha, Nebraska, FCSA is a financial services company that offers equipment leasing, loans, and other related financial services to farmers, ranchers, and rural infrastructure providers. FCSA works with an external vendor that sends them extraction files that they would then store in an old database. Over time, the system changed ownership and after numerous support teams, a black box of data was unintentionally created. As employees came and went, the data and knowledge that they had acquired, left with them. This led to gaps in data, negatively affecting insights and past learnings.
Farm Credit Services of America: BI Obstacles
Like many BI teams, FCSA’s struggles began with the massive amount of data that existed in their enterprise data warehouse. An enormous amount of time was being wasted performing manual searches, as the BI team needed to frequently comb through the enterprise data warehouse’s fields to determine how each was calculated or to find their sources. Perhaps most frustrating for FCSA was the major data-quality issues that arose due to manual, error-prone data processing.
These hurdles were made even more difficult by FCSA’s partner-vendor, which was responsible for after-hours customer data feeds. The vendor was performing a large-scale upgrade on their end and estimated that around 500 fields inside the data warehouse would be deprecated or changed.
Hear from FCSA How Automated Data Lineage and Discovery Accelerated Their Impact Analysis Research
One Day instead of One Hundred
Watch the Webinar
Manual Data Mapping Slows Progress for BI & Analytics Teams
When FCSA’s team heard about the vendor’s new software, they decided it was the perfect opportunity to upgrade their own supporting database and data processes. So, they decided to create an entirely new database.
“We really just looked at this as more of an opportunity, than necessarily a challenge…to be able to create a new database and have a clean load process. We could figure out, in terms of ownership and usage, what was valuable and needed to be reinvented in the new database.”
– Kelly Jenkins, Lead Financial Systems Analyst, FCSA
However, the process of converting their systems proved difficult. Their first step involved manual data mapping, a process that took four months to complete, but there were still plenty of unanswered questions. In order to fully comprehend the impact of changes on any of the reporting or integrated systems, FCSA had to manually go through database view objects, ETL, table objects, and stored procedures. They also had to consider database dependencies specific to the database. Due to the large amounts of manual work involved, they budgeted this to take 100 days to complete.
Automated Data Lineage Helps BI & Analytics Understand Their Data
FCSA needed a better solution as manual data mapping was proving to be too time-consuming and it couldn’t deliver the improvements expected from a technology conversion. So, they began using Octopai’s superior automated data lineage and data discovery tools.
Octopai’s automated data lineage tool helped them locate and isolate the root cause of errors instantly as well as understand the impact of any changes. The automated data discovery tool helped them find which data they were looking for and to identify who owned each data set.
Using Octopai, FCSA’s Lead Financial Systems Analyst was able to hone in on specific resources, which allowed her to establish very concise communication to specific users, thus saving her time and energy. In less than a day, the owners of data were located, which essentially flipped the conversation that she had with business owners. Instead of sending out a widespread message, she could turn to specific people, significantly streamlining the process. With Octopai’s help, she could now say, “Hey, these fields are being deprecated or being altered. I see they are being used here, let’s discuss what the impact will be.”
Instead of relying on the other party to complete the research, the FCSA analysts could now do it by themselves in just a matter of hours, which helped facilitate a sense of independence and confidence. The platform was easy to use for “non-data people” like many in the organization, and enables full visibility to the entire data supply chain, empowering BI and analytics teams to easily find and understand their data so they can enable the business to make better decisions based on solid, qualified data.
Automated Data Lineage & Discovery Provides Enterprise-Wide Benefits
With Octopai’s suite of metadata management tools in place, the BI team had a central location for data lineage. This allowed them to fix issues at their upstream source while in turn, reducing the time required for research. What once had taken multiple analysts weeks/months of manual data mapping, could now be done by a single analyst in just a single day.
“Gone are the days when we would have to ask multiple questions to multiple people, we try to target the root [of the issue] and fix it at the most upstream source so that it flows naturally through the systems. We now get the benefit of proliferated data quality.”
– Andrew Stewardson, Data Risk Manager, FCSA
Some of the key improvements included:
- Discovery and lineage tools enable new employees to work independently
- Faster research processing
- Measurable value from data lineage
- Migrated data checked for usage and duplicates
- User-friendly interfaces allow better interaction with non-data professionals
Octopai saved FCSA weeks of impact analysis research with automation. Think of the wonders it could work for your enterprise.
