Besides the technical aspects, data migration also requires the right tools, processes and stakeholders if it’s going to be done right. Therefore, we’ve put together a top-down guide to the best data migration plan, so you can avoid common mistakes and have the best transition possible.
Planning Pitfalls
Before you begin, you’ll develop a data migration strategy; determining your target destination and the necessary tools otherwise your data could be at risk. This is one of the most overlooked parts of the process because it’s a thankless task. Everyone is excited about choosing a new database platform or a new storage medium. After all, the entire point of data migration is so your business can use newer and better systems. The migration itself is just a process that needs to get done to make the upgrade happen.
For this reason, it can be tempting to delegate as a lower-tier project. But data migration is complex and can cost you significantly if it is delayed or something goes wrong.
Integrating Business and IT
Ideally, you’ll want stakeholders from both the business and the IT sides to be part of the planning. This is a common point of failure, when such migration projects are treated and delegated as a simple project.
However, a significant number of business decisions are to be made during the process. For instance, what type of data are you migrating? If you’re migrating actuarial tables to a new database, for example, IT personnel may not have the expertise to do ad hoc testing. Is the exact formatting central to your company’s operation, or can formatting be changed to make the process easier? Is the system business critical, or can it be shut down overnight? Does all the data need to be migrated? Will processes need to be rebuilt or will new workflows need to be conceived?
Your IT team is going to need to know these things, and if they don’t, your business can suffer costly delays and mistakes.
Define the Scope of Your Project
Depending on what type of data you’re migrating and why you’re doing it, the scope of your project may be very narrow or very broad. For example, a 90-day CCTV archive for a single location should be a simple, easy job. Moving an entire company’s records, on the other hand, can be very involved.
Other considerations come into play regarding governance and if users customize their own settings, or if different users have different types of permissions. This type of data can be difficult to migrate, because the settings on both systems may not have apples-to-apples equivalents. This is another area where IT is going to need input from people on the business side of things.
Finally, you’ll need to decide if you’re doing an all-at-once Big Bang migration, or if you’re migrating your data a bit at a time with a trickle migration. Different migration methods require different approaches, different testing methods, and even different software. Changing direction halfway through the process can force your IT team back to the drawing board, costing you time and money.
Budget, Timeline and Due Diligence
Once you’ve defined the scope of your project, the next step is due diligence. This is another step where the business team and IT team will need to work together. The IT team will need a complete understanding of business issues, both current issues and anticipated future needs, in order to migrate successfully.
You’ll also need to agree on a timeline for the project. Most data migrations take between six months and two years from planning to completion. Be realistic with your expectations and listen to your IT team. They’re the ones in the best position to understand just how long the project will take.
Finally, successful data migration will require an adequate budget. You’ll need to account for personnel, validating the original data, data mapping specifications, coding the migration software, multiple levels of testing, loading the data and validating it afterward. If your data is sensitive—for example, financial or health care records—you’ll also need to verify compliance with applicable regulations and requirements.
Backup Your Data
Before you start the migration process, it’s essential to have backups of the original data. Even the best teams can make a mistake, and even the best hardware can fail. By creating backups first, you’ll ensure you won’t permanently lose any business-critical data. If you’re handling sensitive data, you’ll also need to securely destroy your backups once the migration is complete, just as with the original copies.
Test, Test and Test Some More
To people who are experts in business, and not IT, data migration can seem quite straightforward. After all, you don’t need a team of highly-paid professionals to copy a PowerPoint presentation to a flash drive. Data migration, on an enterprise scale, is more complex—as we hope you’ve already learned.
Because of this, your IT department will need to run a series of tests, starting with small amounts of data and working their way up to entire volumes. With each test, they’ll encounter bugs in the migration process or errors in formatting, all of which will need to be corrected and re-tested. This is the main reason data migration is a long-term process.
Audit Your Results
After the migration is complete, the last step is to validate the results. This can be done in a variety of ways, but broadly speaking there are two methods.
Automated validation will involve a script your team develops during testing. This software will go through the data and ensure all information is in the right fields. It should also include a null check to search for any data that might have been missed. If your users have different permissions, then that will need to be tested as well.
Ad hoc testing means querying the data as you would in a real-world business scenario. For all but the smallest projects, ad hoc testing is prohibitively expensive to do for the entire dataset for every file. However, a thorough ad hoc test should reveal any glaring issues.
