Preparing For A Successful Data Migration

When it comes to data migration, most of the advice you read from the experts sounds like it came from fortune cookies you might receive at a restaurant run by Confucius himself.

It’s easy to give obvious advice, such as “Find the right people who can be trusted,” or “Avoid downtime and minimize risks,” or “Find the right tool that can do everything centrally and automatically.” But concrete and specific suggestions are probably more helpful. Confucius was a sagacious man, but for data migration we’re on our own to discover the wisdom of the ages. When analyzing enterprise-level migration projects we were involved, the success stories share some common characteristics. Most of them can be summed up in the answers to the following questions:

What type of data is being migrated?

It is very important to determine the type of data being migrated since there are disparate migration types.

• NAS-based – where network shares are migrated from one system to another

• LUN-based – where the disks (LUNs) are migrated from one storage to another in block level

• VM-based – where the entire virtual hosts are migrated from one hypervisor to another

• Application-specific – where one set of database or objects, is migrated from one host to another

Also, what type of overall migration is this?

• Local migration – within the same datacenter

• Remote migration – from one datacenter to another

• Cloud Migration – to Cloud provider (e.g., AWS, SoftLayer, Azure, Google Cloud)

What is the current storage environment?

It is equally important to have a reasonable understanding of the current storage environment. Using SAN as an example, what hosts, in terms of OS, applications are involved? What are the FC switches, and RAIDs? Most of all, how large and inter-dependent is the labyrinth of the SAN? Operational complexity could be exponential and not linear relative to the size of the system. Intimate understanding of the picture avoids multiplication of complexity and obviates disasters.

How much data is being migrated?

It cannot be stressed enough that an accurate and clear estimation of the amount of data must be made. Furthermore, the detailed number of hosts, LUNs, and storage systems must be determined. Often, the required effort cannot be judged merely by the terabytes that need migrating. Even knowing the number of LUNs is still only half of the story since the number of hosts may be the biggest factor in determining the time and effort required for cutover after the data is migrated. Of course the amount of data and data rate will determine the absolute time needed for migration, but there are other factors involved as well.

How much time overall will be allocated to the project?

This is almost obvious, but it may not be that simple a question. The amount of data is usually predetermined, so the only other parameter is the data migration rate. This again highlights the importance of understanding the data, application, and environment because knowing if the data can be migrated in a divide-and-conquer manner helps to formulate the strategy – not only to meet the timeframe, but also to control operational complexity, as well as potential cost.

How active is the data and how much downtime is available?

How active the data is determines if the migration can be completed in time, or at all. This immediately raises the question – how much impact to production will be tolerated during the migration? Furthermore, during cutover, how much production downtime will be allowed? Knowing this ahead of time allows agreements and strategies to be worked out well before migration starts, and also helps to ameliorate stress during cutover.

What is the right tool for the migration?

The selection of data migration tool can make or break the project. But what are the evaluation criteria specifically?

Again, using the SAN migration as example, a migration tool should be able to:

1. Minimize the need for looking up every WWPN, host, storage target, and GUID of every LUN, thereby saving hundreds of hours of painstaking, boring, and error-prone work.

2. Install seamlessly with no disruption to the production environment, which significantly reduces risk during preparation and execution of the migration.

3. Provide a clear and intuitive display for the automatically discovered SAN configuration, complete with identities, status, and I/O statistics for all hosts, and LUNs to be migrated (as well as those not to be touched), so the migration operation can be organized correctly and accurately with high confidence.

4. Intelligently and flexibly allow minimizing, or even eliminating impact to production during migration. For example, migrating at non-production hours, or automatically yielding to production traffic, with adjustable intensity/throttling.

5. Monitor, report, and alert on the migration operation in a centralized manner, while also including the production traffic and activities in general with valuable status and statistics.

6. Perform data integrity checking of the intermediate images before cutover occurs, allowing any issues to be discovered before production is switched over to the new storage.

7. Allow scrubbing of the data disks after the data is successfully migrated out, especially when storage is to be returned after lease. The scrub level should be variable to allow an efficient mode for speed, or a secure mode with multi-pass scrub to ensure data is securely erased.

8. Provide a clear, precise and complete post-migration report detailing exactly what happened during the migration process every step of the way, down to individual LUN activities.

9. Demonstrate a solid track record of success and satisfied customers for the given type of migration (through case studies and reference accounts, for example).

Of course the product describes exactly the DMS we have in Cirrus Data. But that is because these are the functions and features we wished for when we were doing data migration using other tools. Since no such product exists, we built it.

Who are the right people for the job?

Now we circle right back to the Confucius wisdom: Above all, find the right people — whether using internal personnel or external professional services partners — who can be trusted and who have experience with the given type of migration. How do you know if they are truly qualified? Well, it is actually very simple: do they articulate the questions posed above with concrete answers, or do they just give you a bunch of fortune cookies?