What tools do you use for the ongoing monitoring of checksums?

At MoMA we store all digital collections materials in the bagit format, and worked with Artefactual Systems to develop a fixity checker that runs on the same server that hosts our Archivematica instance. It checks all AIPs found in the Archivematica Storage Service for validity (which if you're familliar with the bagit format, you'll know, checks aspects other than file fixity - i.e. files that shouldn't be there, files that are missing), and emails any failures to Storage Service admin accounts.

This fixity checker sends metadata about its activity to our browser-based repository management system (also built by the lovely people at Artefactual Systems), so that we can index and retain a permenant record of all fixity checks, allowing reporting in various ways. In the event of a fixity error, our management app allows users to recover the AIP from a manually retrieved offline backup (i.e. an offsite LTO copy).

Here's a screenshot of the fixity audit dashboard widget (faulures are fake!): 

And here is a screenshot of (part of) a full report:

This system is brand new, so I don't have much in the way of lessons learned at scale. All of this is being released by Artefactual Systems (free and open source, naturally).

There are however some major changes we are making to our storage system this year due to an incomming massive influx of collections materials in the area of 2PB. We are moving from a fully disk based storage system, to a heriarchical system that is primarily LTO with a fast disk cache. This will give us an exponentially lower $ cost per TB, lower energy bills, as well as a built-in method for making a third backup for off-site offline storage. This new system is a managed service, and actually provides file-level fixity checking that can be queried via an API. We will be working to get our fixity checker app talking to this API so that we can get the managed service's fixity check metadata into our management app in the same way we get our "own" fixity metadata in. Will will probably rely solely on the managed service's fixity checks, since it has a logic for ensuring things are checked that is more sensitive to the fact that the materials are on tape rather than disk. If you are comfortable with moving away from file level fixity checks there are some proprietary tape formats that have built-in fixity metdata that allows for fixity to be checked without reading the full bit streams (i.e. T10000C) – though that's not what we're doing. We're still big on file-level checks (for now). The kind of fixity Andy is talking about with HDFS is in my mind a given –  or at least should be at any institution where you have an IT department. If your storage appliance is not doing some method of integrity checking (at the filesystem level, or at the appliance level via RAID, etc) then you're in big trouble and have bigger fish to fry than human readable fixity checks. This is a basic requirement of any enterprise-ish storage appliance, not just for "archival" applications.

The AVPS fixity tool serves a great purpose, even in our situation. I'm thinking about using it for monitoring our pre-ingest staging area, where things live prior to ingest into our repository. The stats in the report are pretty great not just for monitoring fixity, but for just keeping a handle on what's going in/out on a file share.

We've developed a checksum monitoring tool called Fixity (http://www.avpreserve.com/tools/fixity/) that will handle many of your requirements.  Fixity will:

• Check for file integrity (MD5 or SHA256) and file attendance (documented in our whitepaper here: http://www.avpreserve.com/wp-content/uploads/2014/07/FixityAndFilesystems.pdf)
• Perform scheduled scans of preservation masters
• Alert users via email of any detected changes (with different users per collection)

As you head up towards a petabyte, you might want to consider switching to HDFS. It's very cost effective, highly resilient against disk failures, and has been proven to work at scale. It has fixity checks built in (IIRC, it defaults to checking every data block once every three weeks), and if data is found to be damaged or lost it automatically heals itself using the checksum metadata and the other copies spread over the cluster (defaults to three copies, over at least two distinct racks).

Any filesystem that goes up to a petabyte will probably include something like this, as HDD failures become fairly frequent when you have hundreds of disks. We have about a petabyte of raw storage, and experience a couple of disk failures per month.

Of course, filesystem-level checks only prevent against storage rot/failure. If you want to ensure data isn't accidentally deleted, you'll need to monitor things at the workflow and/or collection level. Tools like Fixity can help here.

1. Institutions looking to develop a full digital preservation program -- most digital repository software handles ongoing/scheduled checksum verification for master files, but these systems in particular were suggested: