That’s a great point, and one that’s not given enough attention by archive storage vendors — after all, it’s an application problem.

When I was spending more time on SNIA LTACSI, the Long-Term Archive and Compliance Storage Initiative (quite a mouthful), I tried to draw a distinction between what I call “physical readability” and “logical readability”.

Physical readability means that I can get back the bits that I wrote 100 years ago, in the exact order I wrote them, with perfect fidelity. Logical readability means that I can extract the semantic meaning from those bits, the same meaning that they had 100 years ago. The first problem is purely a technical one, and the second one is sadly not.

Physical readability is something that storage vendors can address without making the customer change the way they do business. Permabit Enterprise Archive, for example, allows seamless refresh to new media or even to new kinds of technology simply by adding new nodes to an existing system. Since there’s no core component in the grid architecture to go out of date, the entire system can be piecewise refreshed over time, multiple times.

Logical readability, on the other hand, requires that the customer change the way they handle data. It requires that they adopt widely standardized and widely documented, or self-documenting, data formats. It requires that they consider archiving virtual machine images with critical applications that are no longer supported, and identify how much of the environment must be virtualized to maintain long-term access. And most painfully, it requires that they immediately migrate data from formats that are already unsupported, lest it become more expensive to do so in the future.

Archive is a class of storage, storage that can solve the physical readability issues, but it’s also a set of processes. You can’t implement the storage without a plan to use it as part of a complete archive environment.

Regards,
–Jered

Fair enough — Centera now does support XAM. As a co-chair of the FCAS TWG, I certainly appreciate the support!

Why doesn’t Atmos use XAM?

–Jered

Given the purpose of an archive, my greatest concern is related to point #4, and something you and I discussed in the past.

Assuming the integrity of the bytes on the storage medium are not compromised, and assuming the archive can reconstitute the data (itself an enormous challenge as you pointed out), there’s no guarantee that the data will be in a usable form.

That is to say, the applications or complex systems that originally wrote the data may no longer exist in usable form. I like to use the example of a modern website consisting of HTML, style sheets, various client and server side code, one or more databases, and perhaps data generated by services external to one’s own organization.

One could archive a PDF of some single state (or several states) of the website, or save the collection of files and data to an archive. But one must ask if either of those methods are sufficient to meet the expectations of the business (or regulatory/legal requirements such as the FRCP)? Can the archived data be reassembled in a meaningful way?

To reconstitute a website, beyond a simple reconstitution of its files and data, one would need compatible, operational versions of every application used by the site at the time it was archived, and a means to restore the files and data into those environments. Programming language interpreters/compilers, operating systems, databases, application servers, content management systems, office applications, browsers, etc.

This is equally true for most complex business systems including CMSs, CRMs, ERPs, SCMs, etc. And with the growth in popularity of systems developed using loosely coupled services, I believe the problem will only worsen over time.

The roach motel metaphor falls short. It’s analagous to wearing mittens to build a 50,000 piece puzzle from a box with no picture on the cover to help guide its reconstruction.

Until we find a suitable range of solutions for this challenge, today’s long-term digital archives are (for the most part) bit buckets from which we hope we can retrieve meaningful information in the future.

One addition to your last point: EMC Centera is a product with a proprietary API “AND an industry standard one: XAM”. Centera now supports both.

All the best,
Steve