Digital Preservation Q&A - Recent questions tagged disk-image

How/where to store metadata about optical media sector layout in METS/PREMIS

Mon, 05 Feb 2018 16:09:33 +0000

I'm drafting a METS/PREMIS profile for images/rips of optical media images (ISOs for data sessions; WAV or FLAC files for audio). One of the pieces of metadata I'd like to include is the output of the cd-info tool, which contains information about the sector layout of the disc. Here's an example:

https://gist.github.com/bitsgalore/9a2838481574c040f7c4b7da4ed59926

However I'm unsure how (and where) to store this info in METS. My initial idea was something like this:

Create a METS techMD element which is associated with the structmap div element that encompasses all files that were extracted from the physical disc (typically one ISO image and/or multiple audio files).
Inside this techMD element, create a PREMIS OBJECT instance with xsi:type="premis:representation" (since it describes the disc as a whole, and not an individual ISO image or audio file!)
Then use PREMIS unit 1.5.7 objectCharacteristicsExtension as a container for wrapping the iso-info output.

The problem here is that the PREMIS 3.0 data dictionary says that the objectCharacteristicsExtension unit (and also its parent unit 1.5 objectCharacteristics) is "Not Applicable" for the intellectual Entity and Representation object types!

This makes me wonder how others are handling this. Is this an oversight of PREMIS, or is there some other (possibly
better) way to do this that I've overlooked?

Any suggestions appreciated!

Parity data for ISO images: anyone doing this? Best practices?

Wed, 25 May 2016 13:48:28 +0000

Earlier today I had a discussion with a colleague on possible ways to store (ISO) images of CD-ROMs and DVDs in our repository system. In addition to checksums, he suggested to also generate and store parity data for each image file (e.g. using the par2 tool: http://manpages.ubuntu.com/manpages/trusty/man1/par2.1.html). This would enable one to repair files in case of bit-level corruption.

This made me wonder how commonly parity info is used in digital archives, and if there are any best (or at least recommended) practices. E.g. what levels of redundancy are typically used? I'm not really familiar with this at all, and it's not a subject that is often mentioned in digital preservation discussions (but see https://twitter.com/anjacks0n/status/733281959762395139).

Incomplete ISO image after imaging CD-ROM - how to prevent and detect this?

Thu, 03 Sep 2015 10:33:26 +0000

While running some tests creating CD-ROM ISO images with ddrescue, I ended up with ISO images that were incomplete in some cases (last ~50 MB of image file missing), even though ddrescue’s log file didn’t report any errors. Below the results I got from 4 attempts at imaging the same CD-ROM on the same PC (note that some of the ddrescue options I used are slightly different, but this appears to be unrelated to my issue). For this I used 2 different external DVD readers:

Reader A - modern Samsung USB device;
Reader B - old SATA (internal) device, refurbished to USB.

Attempt 1 - reader A

Command line:

ddrescue -b 2048 -r4 -v /dev/sr0 windows_98_upgrade_nl.iso windows_98_upgrade_nl.log

This resulted in a 601.7 MB ISO image. Here are the contents of the log file:

# Rescue Logfile. Created by GNU ddrescue version 1.17
# Command line: ddrescue -b 2048 -r4 -v /dev/sr0 windows_98_upgrade_nl.iso windows_98_upgrade_nl.log
# current_pos  current_status
0x23DC0000     +
#      pos        size  status
0x00000000  0x23DCB000  +

I.e. the log file indicates the CD was imaged without problems. MD5 checksum is 82603be06a8142aad1dfaa9e1279371f

Attempt 2 - reader B

Command line:

ddrescue -d -n -b 2048 /dev/sr0 windows_98_upgrade.iso windows_98_upgrade.log

Again this resulted in a 601.7 MB ISO image, again with no indication of read errors in the ddrescue log. MD5 checksum was (again) 82603be06a8142aad1dfaa9e1279371f.

Then by chance I discovered some text files in the image file weren’t readable, so I did a third try, now again with reader A.

Attempt 3 - reader A

Command line:

ddrescue -d -n -b 2048 /dev/sr0 windows_98_upgrade_test.iso windows_98_upgrade_test.log

This resulted in a 660.9 MB ISO file. Again no errors in the ddrescue log; MD5 checksum is 24f0f746d0817121253c6b1242d4246e. After mounting the image, the text files that were problematic in the earlier images were normally readable.

Attempt 4 - reader B

Command line:

ddrescue -d -n -b 2048 /dev/sr0 windows_98_upgrade_refurbished_onemoretry.iso windows_98_upgrade_refurbished_onemoretry.log

Result was identical to result of attempt 3!

So summarising, 2 runs of ddrescue (using 2 different USB readers) resulted in exactly the same error, whereas the remaining 2 runs (again using 2 different readers) completed fine. So what’s going on here!?

Md5sum directly on physical CD

As a first step I computed the MD5 checksum directly on the phyical disc, using:

md5sum /dev/sr0

I repeated this 4 times, using both readers A and B, plugging them into different USB slots. In each case the result was 24f0f746d0817121253c6b1242d4246e, which is identical to the hash I got for the ISO in attempts 3 and 4 (confirming these images are correct).

Comparison of ISO images in hex editor

I also did a comparison of the intact and faulty ISOs in a hex editor. This revealed that in the faulty images a block of about 59 MB of data is missing at the end of the file. I double checked this by copying the block of missing data to a separate file (missingblock.dat), after which I appended it to one of the faulty files using:

cat windows_98_upgrade_nl.iso missingblock.dat > isorepaired.iso

Then check:

md5sum isorepaired.iso

Result:

24f0f746d0817121253c6b1242d4246e

Which corresponds to the value of the intact image.

But why is this happening in the first place?!

The really important question is why this is happening in the first place, and if there’s any way to avoid it? The thread below on the ddrescue mailing list describes a somewhat similar (but not quite the same) issue:

https://lists.gnu.org/archive/html/bug-ddrescue/2014-02/msg00003.html

Note the following quote from the response by ddrescue’s main author. He suggests that the problem might an issue with a USB port, adding:

Ddrescue can’t know if the data are really good or if the hardware is lying about it.

If correct, this would apply to other imaging tools as well. Based on my results, I’m curious if other people may have run into similar issues. More importantly: how does one even detect errors like these? Of course it is always possible to run a checksum on the physical medium and then compare it to the ISO checksum, but this takes ages. A more quick and dirty approach would be to compare the size of each created image against the size of input medium. E.g. to get the size of a CD-ROM I can use something like this:

lsblk /dev/sr0 -n -b

Result:

sr0   11:0    1 660850688  0 rom

(Third column is size of CD in bytes).

To get the size of the ISO image:

du -b windows_98_upgrade_refurbished_onemoretry.iso

Result:

660850688   windows_98_upgrade_refurbished_onemoretry.iso

This does not guarantee the image is correct, but it will detect missing blocks of data.

I also ran some cursory checks with isovfy and isoinfo, but the output of those tools turned out to be identical for both faulty and intact images, so they’re probably not very helpful for this sort of error.

I’m curious how other people/memory institutions are dealing with this. Any thoughts / suggestions are welcome!

Addition

On Twitter Alexander Duryee rightly pointed out that a CD-Rom's Primary Volume Descriptor contains a field with the size of the disk (this is also where lsblk gets this value). So one would assume that ddrescue would check against this number. Apparently it doesn't do this, so I think I'll reprt this as a bug. (Note that such a check doesn't guarantee the copied data are identical to the source disc.)

How to detect Hybrid ISOs?

Tue, 04 Nov 2014 15:37:39 +0000

Asking for a friend:

How does one detect a hybrid file system ISO, the ones that have HFS and Joliet for example?

$ head -c 2048 my.iso | grep -c Apple_partition_map

Is there any tool that would give more meaningful metadata than "ISO9660'?

When to redact AIPs vs DIPs in digital archives workflows?

Tue, 24 Jun 2014 14:22:05 +0000

Tools like bitcurator are integrating processes for redacting information from disk images. So, not just identifying information that you might want to use to make aprasial decisions about what to keep but also the ability to directly redact information from files. I know some folks are planning on using those tools that auto locate personal info and then redacting files or redacting information from inside files.

Given that we have (at least in theory) Archival Information Packages and Disimenation Information packages we could be redacting, when should archivists be redacting the archival copy and when do we want to just be redacting the access materials? If one has information that is sensitive now but would be useful and non-sensitive in 50 years? Or, is the threat of mantaining sensitive information and multiple copies of materials too significant to warrent such an approach? Or, is this something that is going to depend on the particular issues in a collection. If the answer is "it depends" what is it that it depends on?