Migrating Our Own Data – With A Twist

As Cirrus Data grows, our own storage demands continue to increase. Recently our company was at a point where our current storage was nearing its maximum capacity, and the multiple servers that were fighting for the limited number of spindles were experiencing performance degradation. It may sound a bit embarrassing, but until recently we used the same aging storage system – comprised of older EMC Clariions and other arrays – for both production and testing. Such is life at a start-up. Contributing to the performance and capacity issues was the fact that the test storage was being stressed even further as the scale of our testing increased substantially – more like exponentially. We finally reached a point where we really needed to separate the production data repository from the testing storage.

This meant that we had look into migrating our own data.

We happened to have a perfect tool for this purpose — our own DMS, which stands for Data Migration Server. It was also fortuitous that we obtained some licenses for IPStor, the storage product that was one of the fruits of our past labor at FalconStor. Using the IPStor Java Console again brought back a lot of great memories. Our team had implemented such ingenious design and paid such attention to detail that the product still reveals a degree of sophistication no other storage product can match even today. I know this because I have continued to keep a watchful eye on the storage industry since leaving FalconStor. We placed a few large SAS disks into a generic appliance, and a few minutes later the system was ready for production, with snapshots being replicated to a remote unit (located in Wayne’s basement). The plan was to further replicate to our cloud storage, a little bit farther away from Wayne’s basement, like a few hundred miles.

Once the new IPStor system was ready, we assigned the new storage by LUN Masking from IPStor to the DMS appliance we were going to use to migrate the data. We then simply inserted DMS into the FC ports of the old storage system (using only one node, since we were confident in our own technology). Without any interruption to our production, data started to migrate immediately. Since we really did not have that much data, this process would take approximately 24 hours to complete. So the only thing left to do at this point was to wait a day until the migration was complete so we could cut over to the new storage.

The events that unfolded next sound like I made it up just for dramatic effect, but I have not. It is a true story. As luck would have it, our building’s power went down that night, killing all the servers in the lab. No, we did not have supplementary power, and our UPS does not last that long. Naturally the migration session was interrupted, twelve hours into the migration. The question was, did we just waste twelve hours and have to restart the migration from scratch? You have to understand that for live data migration, our DMS has to track all changed data in real time. This means it needs to record the position of the changed data without affecting any client data I/O performance, even if an ungraceful shutdown of the appliances is encountered (such as the power outage that occurred in this case). Essentially, this power loss was a true real-life test of our own product.

Of course we passed this unintended test with flying colors. Once the power was restored, we simply resumed the migration process, which continued pretty much where it left off, successfully completing the rest of the migration. We cut over the storage as scheduled. Unfortunately, restoring the rest of the infrastructure did not go quite as smoothly. Since all the VMs were shut down ungracefully, it took hours before things were back to normal.

Naturally, we have tested this kind of scenario countless times to ensure DMS can continue smoothly should any unexpected disruption occur for any reason. Our enterprise customers all over the world have used our DMS to successfully migrate massive amounts of data. Still, there is probably no better way to increase our confidence in our own products than by using them for our own operations, where our own survival is at stake. Some people call it “eating your own dog food.” More cultivated people tend to use more refined language, like “tasting your own medicine.” But I am not sure which is worse – dog food or medicine.