Moving house.
If those words didn’t cause you to shudder, you’re a big exception – and also exceptionally lucky. Moving is generally a high-stress event for multiple reasons:
- Finding a reasonable mortgage offer and getting approved for it
- Concern that you won’t have enough money to cover unexpected expenses
- Packing up everything you own… and then unpacking it
- Hoping all your possessions make it to their destination without getting broken or lost
We’re not sure if data assets experience stress when they have to migrate to a new residence, but certainly all of the humans involved in the data migration process do!
While the financial stress of migration is usually shouldered by company executives, the entire data team shares the stress of packing up data assets, moving them, unpacking them and putting them in the right locations, and hoping everything got there in one piece, with nothing broken or lost.
Making moving easier
What’s a time-tested strategy to ease the packing-and-unpacking part of moving house? Making a moving map.
A moving map (or the same thing by any other name) is a planning process: identify a specific group of possessions that were in one location in your old house, and designate where they will go in your new house.
Sometimes it’s simple: all the clothing from your master bedroom in your old house should go to the master bedroom in the new house. In both houses the master bedroom is the only room with an attached bathroom, making its identity quite obvious.
Sometimes it gets a little more complex: your two daughters, Amy and Cindy, were sharing a room in the old house, but they’ll each have their own room in your new house. You would like all of Amy’s books, toys and clothing from the old girls’ room to be put directly into her designated new room at the new house. Same goes for Cindy’s possessions.
Sometimes the level of complexity boggles the mind: have you ever organized the move of a Fortune 500 company’s offices to a new location following downsizing and the restructuring of the company’s hierarchy?
Well, don’t.
In any event of a move, move mapping can facilitate an easier, smoother transition, ensuring everything gets where it’s supposed to go.
Move mapping for data
When data moves house, it’s called “migration,” which conjures up images of birds flying south for the winter, naturally and instinctually.
We wish.
Data migration is anything but natural and instinctual.
But where instincts fail, a thought-out plan can prevail. Let’s take a look at the data mapping steps that you can use to create a thought-out migration plan and detailed migration instructions.
The 5 data mapping steps for a smooth data migration
1. See what source data you really need to move
What do you do before packing up your house for a move, if you’re smart? You go through your possessions and decide which items you actually need and should come with you, and which you don’t and should be given away or discarded. (Well, that’s until it gets too close to Moving Day. Then you just throw things into boxes like crazy.)
Bringing unneeded items with you will make packing take longer, make getting settled take longer and likely make your move more expensive – needlessly.
Moving data is no different. A smart data mapping methodology starts with taking stock of your data assets and determining what really needs to come with you.
What should stay behind?
- Duplicate data
- Outdated data
- Redundant data
- Irrelevant data
How do you quickly identify data that fits into these categories? After all, if it takes too long, you’re likely to run out of time and start throwing things into boxes like crazy.
An automated data lineage solution can quickly and easily create a map of all your current data assets and their journeys through your data landscape, including what systems they pass through.
Spot a dataset created by an analyst who didn’t realize that someone in a different department had created the exact same thing? One of those can go.
Notice a dataset populated from a marketing tool you’ve long since stopped using? Chuck it.
Find some training datasets from an early (too early) venture into AI modeling? Scrap them.
Now that you’re feeling free as a bird without all that extraneous data (or heaving a sigh of nostalgia for a particularly sentimental dataset), it’s time to take all the keepers and start the data mapping process.
2. Identify the data structure of source and target data
Data is the man of a thousand faces.
Structured. Unstructured. Semi-structured. Arrays. Lists. Trees. Text. String. JSON. XML. CSV.
And that is not all. Oh no, that is not all.
The first concrete step in data mapping is to take a data asset slated for migration and identify its structure, as it is currently sitting in your systems. Next, take a look at where you want it to end up. What structure will it need to fit into over there?
If the structure is one and the same, hooray! You got off easy. If not (and it’s probably not), move on to the next step.
3. Identify process needed to transform one structure to the other
How do you turn a JSON file into XML? How do you take a customer prospects list full of duplicates and dead ends and turn it into a clean, solid list of leads?
It all happens through the magic of data transformation. Data transformation can include:
- Aggregation
- Discretization
- Generalization
- Conversion
- Normalization
- Filtering
- Smoothing
Identify which transformation process is needed to take your data from the source structure to the target structure. Then…
4. Create the “map”: the processing instructions
It’s time to create that moving map: the one that your movers will follow to pack up a box full of possessions in your old house, transport it to the target room in your new house and unpack it so everything is all organized and just waiting for you to step in and start enjoying your new home. (Don’t you wish real movers did that?)
This data mapping step is just organizing the information you collected in the previous steps, often with the aid of a data mapping tool.
5. Test the process
Golden rule of data migrations: You can’t be sure it will work until it works.
Test your newly created map on a small test segment of source data. Did it work? Are the results in the right place, in the right structure?
If so, give yourself a pat on the back… and then get back to work.
Rinse and repeat for every other data asset you’re going to be migrating. (Yes, we know we just crammed weeks or months of work into a sentence.)
Ready, set, migrate!
When your complete data mapping document has been tested and finalized, it’s time to move forward with your migration.
This is another opportune time to leverage the automated data lineage system that was so helpful when deciding what data was valuable enough to be migrated. Automated data lineage can keep track of where all data assets are as you progress in the migration; it can also check that the right pipelines have been established.
Data lineage visualizations make it easy and intuitive to see where your target system is holding in relation to the migration plan. And automated data lineage will automatically run periodic scans and update the visualizations as the migration progresses.
Follow these data migration data mapping steps, and your data won’t have to dread a change in address.
Happy moving!
