Based in Omaha, Nebraska, Farm Credit Services of America (FCSAmerica) is a financial services company offering loans, crop insurance, equipment leasing, and other related financial services to farmers and ranchers, agricultural-related cooperatives and agribusinesses, rural home buyers, and rural infrastructure providers.
FCSAmerica works with an external vendor that sends the company a series of extraction files to be ingested and maintained. Over time, internal ownership of the data system had shifted through many support teams. Gaps in knowledge emerged as duties shifted or people left the company. This is of course problematic for any data team.
Recently, the vendor upgraded its application and informed FCSAmerica that an estimated 500 fields would be deprecated or changed. That’s a lot of fields to contend with.
The team realized this was an opportunity to upgrade the supporting database and data processes and create new and shared knowledge that would be rooted in ongoing documentation and support.
The team broke the work down into three phases, studying the overall data impact of each phase before moving forward with any work.
During phase one and two, FCSAmerica had limited capability to scan technical metadata and optimize research. There were options to script some metadata queries to identify data touchpoints and dependencies. But beyond that, the options were constrained. The team took the approach of broadcasting the known data changes to all potential users, who were then asked to independently research specific impacts. It took several months to gather input, define work, and execute, while also feeling confident in doing so.
But why did it take them so long?
Well, in order to understand the impact of those changes on any reporting or other integrated systems, the team had to manually go through its ETL processes, database view objects, table objects and stored procedures and look at database dependencies native to the database. Then the team had to mentally stitch them together and put everything on a spreadsheet to send out to a large group of potential users to gauge the potential impact of these changes. The team asked each and every potential user how or if he/she used each particular field, and the impact of possibly removing or changing the field. The team, of course, had to wait for the flood of responses to come in and spend more time reconciling them.
Kelly, a System Analyst, joined the organization before the beginning phases of a data migration effort, and was asked to continue the research and requirements for the remainder of the project. Shortly before she joined, FCSAmerica implemented Octopai with encouragement to leverage the features for data lineage-related research. Kelly quickly began phase three impact studies using Octopai’s automated data lineage and discovery and was able to complete the remaining impact analysis, about 100 fields or so, in a day.
A single day!
Using Octopai, Kelly was able to target her research to specific resources for the questions she needed answers to. This enabled her to send out very concise communications to specific users such as, “Hey, I noticed you’re using this field for this specific report. The vendor is going to be removing that field or making a change to it, and you have X amount of time to make edits and refactor that.”
Instead of seeking input from a broad audience, many of whom might not look at the communication until one of their reports broke, Octopai significantly streamlined the process.
What used to take months, takes a day with automated data lineage & discovery.
Kelly recognized the self-service nature of Octopai right away and how it minimized the dependency on other people. With the technology’s intuitive features such as automated discovery, lineage, and global searchability she was able to get started immediately and completed the research in 1 day! This was a tremendous value add in her role, making lighter work for the research which led to more time being spent on development activities. Annualized, this saved the organization roughly 100 days of data research.
This is what the Farm Credit data evolution and transformation is all about.
From months to a day.
Providing unprecedented visibility and trust into the most complex data environments.
