Don’t Try This At Home

When we tell people about our Transparent Datapath Intercept, or TDI, their first reaction is almost always “It is not possible.” But after we describe how it works, their comments change to “Wow that is really clever…” Then after we demonstrate it using our products, they say “It is magical!” We promote TDI with the line “The Magic behind the Technology,” but really, there is no magic. It is a simple idea. A bold idea, but it is a simple idea nonetheless.

So simple, in fact, people would say “I wish I had thought of that.”

Thinking of something is only one aspect of solving a problem, however. Sweating out details of the intricate design, then realizing the actual logic in C code is quite another.

Once the TDI is inserted to the actual storage link, it has to handle every I/O command and the corresponding data through that fibre channel connection. At the same time it has to be totally transparent to the environment. We call this a “virtual cable.” From the perspective of the client initiator, there are no changes to the environment – it sees the exact same target ports as before TDI was inserted. From the perspective of the target port, it sees the exact same initiators.

Now let’s just take a peek under the hood, and see what is demanded to pull off such a feat.

First, everything we do must be within the confines of the fibre channel architecture, which means we must live inside the firmware specifications of the fibre channel host adapter. There is only a limited set of programming functions provided. Therefore many times clever schemes must be invented to make up for any deficiency in available tools.

Second, I/O command timing in fibre channel connectivity is a very sensitive issue. Therefore no matter what it takes to complete the commands, the TDI has to do everything, including throwing some candies to assuage the client hosts should any necessary delays are encountered. Otherwise the impatient clients will start bombarding the target with aborts and resets, which can cause an avalanche that will jam up the link to I-495 Long Island Expressway during rush hour.

Third, standards are a good thing and everyone loves them. That is why everyone has their own, and the storage world is no exception. While everything is supposed to be compatible under the fibre channel and SCSI protocols, idiosyncrasies are everywhere, especially due to legacy systems. But legacy systems are most likely to be what we will encounter – migrating data out from old storage to new.

… And too many more points to list here…

Finally, the TDI is not used simply to show off we can transparently insert into a data path. We actually intercept for specific purposes. We have to interact, or even handle some of the I/O’s and data. That means we have to be able to intelligently process the standard commands, and handle some not-so-standard ones. While many commands can be just passed through, some subtle ones cannot. For example, how to you deal with SCSI reserve/release for cluster operation when the data is cached?

What makes TDI work is the vast amount of knowledge and experience from the CDS engineering team, combined with countless hours of analysis, design, and testing of the technology. Special programs had to be written just for tracing the code paths to ensure the designed logic was validated for various specific conditions. In addition, specific behaviors and conditions had to be manufactured for testing because they don’t normally happen. This included both upstream and downstream, from the client hosts to the storage systems. Both client initiators and storage target simulators had to be built to emulate error conditions. Nothing was left to chance.

There’s more.

CDS software modules perform highly sophisticated operations which interact with complex environments and interfaces with other vendors. Of course our software has no bugs (“ahem…”), but what if there is some hiccup in these interactions that causes the upper layer to crash? TDI must be designed in such a way that it can always pass through I/O regardless what the upper layer modules encounter. It may abandon the advanced functions the products provide, but I/O paths must be kept alive and unhindered as best we can to minimizing the risk of affecting production. A fail-safe mechanism is a necessary component of the architecture.

Yes TDI looks simple – deceptively simple. Many of our users are beguiled by the ostensible simplicity, and start to casually insert the nexus into all the paths at the same time in their production environment. That is when we have to scream “STOP!!!” and ask, have you performed the mandatory preflight checklist? Have you planned out the exact steps? Like, insert one nexus, then confirm the I/O through the nexus, then move to the next. Of course we make it very simple to follow the procedure with our Insertion Wizard which guides the insertion process step-by-step using specific instructions, graphical illustration, warnings for any mistake made, followed by a display of I/O status and statistics for each link successfully inserted.

In short, the magic of TDI lies inside the ingenuity, innovation, experiences, and monumental amount of hard work from the world class storage experts in CDS.

As the disclaimer for TV commercials goes: “The stunts are performed by professionals. Don’t try this at home.”