Digital Preservation Q&A - Recent questions and answers

Answered: What are current best practices for acquiring & preserving Google Docs?

Wed, 29 May 2024 14:24:48 +0000

rclone can mass download Google Docs and Spreadsheets and do convert them into Microsoft Office or Libre Office formats on the fly, see https://rclone.org/drive/#import-export-of-google-documents

Of course that will loose the edit history, but you will get a reasonable current snapshot of a document and can perhaps regularly download to track development over time.

Answered: What is a good tool for normalizing image file formats?

Wed, 31 Jan 2024 14:35:22 +0000

Depending on the file format(s) you're dealing with and whether you're comfortable with using command-line tools, ImageMagick might be worth a look:

https://imagemagick.org/

Don't use this if you're using JP2/JPEG2000 though, as the software library ImageMagick is using for this (OpenJPEG) has some serious limitations that might result in losing things like ICC profiles and resolution information. On top of that I think JP2K support has been disabled by default for several years anyway, because of security issues in this library.

Answered: What are the most effective ways to minimize latency issues in geographically distributed collections?

Fri, 04 Nov 2022 14:22:05 +0000

Latency issues can only be addressed by either improving the connections in between locations or using local caches.

Personally I have found that minio is a good tool if remote data is hosted in object storage. Minio can act as a local object storage proxy that keeps frequently requested data locally available. The tools talking to minio don't get to know if the requested data was locally cached or fetched from remote, but if the cache hits of course it can deliver significantly faster. It can be set up to use a maximum amount of disk space etc.

Answered: Does anyone have a procedure in place for redacting selected content (e.g. PII) from files that they can share?

Thu, 06 Oct 2022 11:59:41 +0000

I was just reminded about this site, and this is a horrendously late answer to this question, but it's a good question and comes up often.

The advice that stuck with me was from my time at TNA, UK, and the most simple approach is to re-save your file as a simple image format and redact the information with black rectangles in the image editor. You'd then re-save the file without layers, so BMP is a good file format for this. I am reasonably sure JPEG would be just fine as well.

You could then recreate a PDF in acrobat with the new images if need be.

The National Archives UK have a redaction toolkit which goes into far more detail and may be of interest to the upvoters of this post.

https://cdn.nationalarchives.gov.uk/documents/information-management/redaction_toolkit.pdf

Answered: How to access a subset of Web Archive pages from a vintage web browser on vintage hardware?

Mon, 14 Feb 2022 13:37:00 +0000

Get yourself a pywb proxy and set it up like this (config.yaml):

framed_replay: false
enable_cdx_api: true
enable_http_proxy: true
enable_coll_info: false
proxy:
    cookie_resolver: false
    coll: proxy
    enable_client_rewrite: false
    enable_banner: false
    default_timestamp: '19980908120000'

collections:
    proxy:
        sequence:
            - index_group:
                ia: memento+http://web.archive.org/web/
            - index_group:
                loc: memento+http://webarchive.loc.gov/all/
                uk_na: memento+http://webarchive.nationalarchives.gov.uk/
                pt_wa: memento+http://arquivo.pt/wayback/

Then connect your legacy browser to the service on port 8080 or whatever other port you're running pywb on.

Pywb will act as a proxy to the web archives listed in "sequence" and deliver the un-rewritter, original raw data via memento protocol. The default_timestamp is the time web archives are asked for resources. The first index_group is just the Internet Archive. The second lists the Library of Congress, the UK National Archives, and the Portuguese Web Archive in cas Internet Archive doesn't have a resource. Like this you can build your own oldweb.today so to say, with real hardware.

Answered: What is a good tool to create a graphical map of a web site (sitemap)?

Mon, 22 Nov 2021 03:07:59 +0000

This one looks promising and is open source:

https://www.astuteo.com/slickmap/

https://github.com/astuteo/slickmap

Answered: Tools to accession via file transfer

Fri, 09 Jul 2021 14:48:33 +0000

Wormhole is great for packages under 10GB (which is twice WeTransfer's limit) since it requires no account and is browser-based.

Answered: Best practices for digitising tapes captured at a low resolution

Wed, 30 Oct 2019 00:04:23 +0000

Hi, for VHS do you mean things like long play or EP? What do you mean by low resolution miniDV tapes? I am not aware of a low-res format for DV.

Anyhow, assuming that you mean things like LP and EP for VHS - I would use the same settings as for regular VHS or BetaSP or any tape format where you can't migrate the signal with the same format as the source (like DV) - Uncompressed or Lossless video (I like FFV1 if you want to save some space over uncompressed) using 720x576 for PAL (720x486 for NTSC) with PCM audio.

If a source tape has a weak signal, you should do your very best to capture as much of the data as possible, rather than damaging it even further by using lossy compression or lower resolution.

Answered: Preserving VCD's, what is the best format

Fri, 13 Sep 2019 12:38:16 +0000

BIN/CUE seems like an obvious choice for this type of disc. From your question I understand that access is the main issue. If so, I did manage to access BIN/CUE images of Photo CDs (which have a similar layout as VCDs) with CDEmu (https://cdemu.sourceforge.io). This creates a virtual drive that you can access with any suitable media player (e.g. VLC), just like any ordinary physical drive. If anything this removes any dependency on your media player's to deal with the image format natively.

See also the notes here: https://github.com/KBNLresearch/iromlab/issues/93#issuecomment-499073161

Note that I haven't tested this with any actual VCDs because I've been unable to locate any in our collection.

Answered: External Write Blocker that works with an External Optical Drive?

Thu, 12 Sep 2019 21:40:01 +0000

The optical media workflows I've come across do not use a write blocker as writing (burning) to optical media is much more difficult to do incidentally than with magnetic or mass storage media. If you absolutely require write blocking for optical media, NYU appear to use write blocked drives according to https://archive.nyu.edu/bitstream/2451/43877/2/Schweikert_OpticalMediaPreservationNYU_2018.pdf. You can also disable disc burining in the Windows registry https://www.technipages.com/enable-disable-cd-burning-windows

Answered: Is checking fixity at the folder level sufficient for knowing if any files within that folder that been altered?

Thu, 12 Sep 2019 21:39:56 +0000

Yes, depending on the method used for computing the checksums and what types of changes you are trying to identify. One solution is to create a tempory archive of the folder and compute the checksum on that file. The following command, using standard GNU/Linux tools will compute a checksum and can be used to identify any change to the files' content, name, persmission, and/or directory structure.

$ tar -cf - somedir | md5sum

This uses the tar command to create a single archive file for the folder and its contents. The c option creates the archive, the f option specifies a file to save the archive to, the - sets the saved-to file as stdout (temporary), the | character directs the created archive to the next command, md5sum, which computes the checksum.

One caveat, this will not indicate which specific files changed (for that a tool like md5deep can be used). Source and additional options/discussion: https://unix.stackexchange.com/questions/35832/how-do-i-get-the-md5-sum-of-a-directorys-contents-as-one-sum

Answered: Best way to read SCSI tape drive on a modern PC?

Thu, 12 Sep 2019 21:39:47 +0000

I can confirm success using option 1, SCSI to USB adapter, for reading SDLT tapes. Our setup includes HP Storageworks SDLT 320 tape drive, Microtech SCSI-HD50 SCSI Bridge (SCSI to USB adapter), and modern Linux (Ubuntu 18.04 and Bit Curator 14.04 both tested).

With this setup, using mt (to control the tape drive) and tar copy the data has been successful.

Answered: Does splitting large master files (temporarily) go against best practices?

Wed, 11 Sep 2019 18:30:31 +0000

As a temporary process this should not be a concern provided you are able to check the reconstituted files' integrity post reconstitution.

Temporary splitting of files to simplify processing often happens in the background with system copying processes anyway, and all data over the internet is sent as packets then reconstituted into files at the end of the copying process.

Answered: Parity data for ISO images: anyone doing this? Best practices?

Wed, 11 Sep 2019 16:42:49 +0000

Thought I should try to give an actual answer here and not leave it for more years.

I don't know of anyone in the digital preservation community that is doing this and I suspect Andy's comment speaks to why. So I also don't know of any best practices. It's not the same as saying that absolutely nobody is but it seems likely that nobody is so hopefully answers your question.

Answered: Do you have any LTFS Tape Library Recommendations?

Thu, 04 Apr 2019 07:52:47 +0000

I know it is a bit late, but I came across this and just in case it helps anyone else here is what I know.

Full disclosure, I work for Magnext which sells tape drive libraries and sells their own brand MagStor which I put together and make software for. I will try to be honest though : )

Off the top of my head...

If you use tape for backups, do you use LTO? If you use LTO, do you use LTFS or a different standard/format?

LTO is the only option. You could use other things, but you would find support in the future difficult.

What kind of tape drive/library set up do you have?

The setup you need depends on your requirements. If you only have a couple 100 TB of data and you can access it all from one computer, you could get a single SAS drive and have someone manually load and move files to the tapes. This would only require a few tapes and the transfer would take about a day per tape. May I recommend our MagStor SAS drive (shameless plug)

If you need to be able to easily push and pull data from multiple tapes from many different computers, at this point you would need a library connected to your network. Setting this sort of thing up can be difficult without someone experienced in IT or assistance from the people you are buying the hardware from. The most important thing here is that you have a libary that works well. Most of the parts that make up the libraries are actually the same and the only difference is the firmware installed on them. If that was not the case, it would be prohibitively difficult for us to repair them. This also means that there is not much difference between which one you get, but I could ask our technicians which they prefer if you want an expert opinion on that.

What software do you use to manage your hardware?

LTFS Software recommendations
MAC: myLTO
PC: PreRoll Post and XenData6 work station

I don't have as much experience with the software side of things, but I hope that helps some.

LTO is the standard when it comes to tape, but as it stands right now the newest version of the tapes, LTO-8 cannot be sold because of a legal battle between Sony and Fuji Film. Once they can be sold again which will hopefully be very soon, LTO-8 is a good choice because it is 12TB for ~$150. (kind of guessing since they cannot be sold right now)

All LTO drives can read up to two generations back and read one generation back. The only drive that breaks this standard is LTO-8 drives which can only read and write one generation back due to a physical change in the tapes.

I myself have not had much experience using LTFS on a PC as apposed to mac, but there is plenty of different software out there to get you up and running on whatever platform you choose, though you will likely have to pay a monthly fee.

I am sure I have missed a lot so If you have any more questions, feel free to ask and I will do my best to answer. Tape is complicated, but it is also the most economical and longest lasting option. Good luck!

Answered: Is there any best practice/guidance on where to store, in METS, metadata about digitisation/digital creation processes?

Mon, 17 Sep 2018 09:01:09 +0000

It might be a bit late to answer this, but at the National Library of Luxembourg we have been using METS/ALTO for digitization projects for more than 10 years. All the technical metadata about and resulting from the digitization project is stored in MIX schema elements within a techMD section, as shown below:

<mix:mix xmlns:mix="http://www.loc.gov/mix/">

<mix:BasicImageParameters>

<mix:Format>

<mix:MIMEType>image/tiff</mix:MIMEType>

<mix:ByteOrder>little-endian</mix:ByteOrder>

<mix:Compression>

<mix:CompressionScheme>1</mix:CompressionScheme>

</mix:Compression>

<mix:PhotometricInterpretation>

<mix:ColorSpace>1</mix:ColorSpace>

</mix:PhotometricInterpretation>

</mix:Format>

<mix:File>

<mix:ImageIdentifier imageIdentifierLocation="//Dss2/d/docworks/EXPORT/bnl_lw/1900-09-10_01/tif">1900-09-10_01-00004.tif</mix:ImageIdentifier>

<mix:FileSize>25003209</mix:FileSize>

<mix:Orientation>1</mix:Orientation>

<mix:DisplayOrientation>1</mix:DisplayOrientation>

</mix:File>

</mix:BasicImageParameters>

<mix:ImageCreation>

...

<mix:ScanningSystemCapture>

<mix:ScanningSystemHardware>

<mix:ScannerManufacturer>ASSY.SA</mix:ScannerManufacturer>

<mix:ScannerModel>

<mix:ScannerModelName>Digitizing line</mix:ScannerModelName>

<mix:ScannerModelNumber>DL 3000</mix:ScannerModelNumber>

<mix:ScannerModelSerialNo>2006 11 25</mix:ScannerModelSerialNo>

</mix:ScannerModel>

</mix:ScanningSystemHardware>

<mix:ScanningSystemSoftware>

<mix:ScanningSoftware>LIBFORMAT (c) Pierre-e Gougelet + Page improver</mix:ScanningSoftware>

<mix:ScanningSoftwareVersionNo>1.0.2519.16684</mix:ScanningSoftwareVersionNo>

</mix:ScanningSystemSoftware>

<mix:ScannerCaptureSettings>

<mix:PixelSize>0.0846666</mix:PixelSize>

<mix:PhysScanResolution>

<mix:XphysScanResolution>300</mix:XphysScanResolution>

<mix:YphysScanResolution>300</mix:YphysScanResolution>

</mix:PhysScanResolution>

</mix:ScannerCaptureSettings>

</mix:ScanningSystemCapture>

</mix:ImageCreation>

<mix:ImagingPerformanceAssessment>

...

</mix:ImagingPerformanceAssessment>

...

</mix:mix>

</xmlData>

</mdWrap>

</techMD>

</amdSec>

A complete METS/ALTO sample package can be found on our tender information page: downloads.bnl.lu/tend2018.html

Answered: Best way to crawl website from localhost with wget, preserving all files in source directory

Tue, 10 Jul 2018 16:19:03 +0000

After getting in touch with pywb author Ilya Kreymer about this, Ilya pointed out pywb's behaviour in this case is triggered by the fact that the generate URL list in this case doesn't contain an exact match for the root URL (http://www.nl-menu.nl) The CD-ROM from which the files were extracted has no index.htm(l) at the domain root level level, which I suppose is quite unusual. This causes pywb to do a prefix query which in this case results in 80k URLs.

Ilya's suggestion was to explicitly add the domain root to the URL list:

echo "http://www.nl-menu.nl/" > urls.txt

On top of that I found out I also needed to add entries for the roots of the Dutch and English language home pages:

echo "http://www.nl-menu.nl/nlmenu.nl/" >> urls.txt
echo "http://www.nl-menu.nl/nlmenu.en/" >> urls.txt

Then we can add the remaining files (and rewrite file paths as URLs)

find /var/www/www.nl-menu.nl -type f \
    | sed -e 's/\/var\/www\//http:\/\//g' > urls.txt

Finally run wget:

wget --page-requisites \
    --warc-file="nl-menu" \
    --warc-cdx \
    --output-file="nl-menu.log" \
    --input-file=urls.txt

This results in a WARC that is both complete and renders in pywb!

Answered: Merging & Deduping WARC files

Fri, 08 Jun 2018 11:17:22 +0000

for merging, warccat concat does the job.

for cleaning/dedup: using JWAT-Tools you can unpack a warc file

jwattools.sh unpack your-archive.warc.gz

the result are single files containing a warc record. then you can remove duplicates or unwanted things and reconstruct the warc just with cat

cat your-archive.warc.{1,6,10} >your-new-new.warc

update: https://github.com/tari/warcdedupe

Answered: Digital Preservation - Server Side

Tue, 01 May 2018 13:33:03 +0000

Sometime ago I migrated a physical server machine to emulated hardware. I added some documentation of the process on an OPF blog post here. With Emulation as a Service (EaaS) now very useable you may want to consider following a similar approach. The research team at the University of Freiburg behind EaaS have also been pursing improved features for preserving acess to linked servers and front-ends using the same backend emulation infrastructure and may have some additional ideas that could help with your use case.

Answered: How/where to store metadata about optical media sector layout in METS/PREMIS

Thu, 08 Feb 2018 17:33:19 +0000

So here's a little update:in the end I serialized the cd-info output to XML, and embedded that in a METS techMD/mdWrap element. Here's an example for an 'enhanced' CD with 2 sessions that contain 20 audio tracks and 1 data track, respectively:

    <mets:techMD ID="techMD_22">
      <mets:mdWrap MIMETYPE="text/xml" MDTYPE="OTHER" OTHERMDTYPE="cd-info output">
        <mets:xmlData>
          <cd-info:cd-info>
            <cd-info:trackList>
              <cd-info:track>
                <cd-info:trackNumber>1</cd-info:trackNumber>
                <cd-info:MSF>00:02:00</cd-info:MSF>
                <cd-info:LSN>000000</cd-info:LSN>
                <cd-info:type>audio</cd-info:type>
              </cd-info:track>
              <cd-info:track>
                <cd-info:trackNumber>2</cd-info:trackNumber>
                <cd-info:MSF>01:22:02</cd-info:MSF>
                <cd-info:LSN>006002</cd-info:LSN>
                <cd-info:type>audio</cd-info:type>
              </cd-info:track>
              ::
              ::
              <cd-info:track>
                <cd-info:trackNumber>20</cd-info:trackNumber>
                <cd-info:MSF>55:23:38</cd-info:MSF>
                <cd-info:LSN>249113</cd-info:LSN>
                <cd-info:type>audio</cd-info:type>
              </cd-info:track>
              <cd-info:track>
                <cd-info:trackNumber>21</cd-info:trackNumber>
                <cd-info:MSF>59:41:40</cd-info:MSF>
                <cd-info:LSN>268465</cd-info:LSN>
                <cd-info:type>data</cd-info:type>
              </cd-info:track>
              <cd-info:track>
                <cd-info:trackNumber>170</cd-info:trackNumber>
                <cd-info:MSF>63:08:25</cd-info:MSF>
                <cd-info:LSN>283975</cd-info:LSN>
                <cd-info:type>leadout</cd-info:type>
              </cd-info:track>
            </cd-info:trackList>
            <cd-info:analysisReport>
              <cd-info:cdExtra>True</cd-info:cdExtra>
              <cd-info:multiSession>True</cd-info:multiSession>
              <cd-info:mixedMode>False</cd-info:mixedMode>
              <cd-info:fullReport>No CD-TEXT on Disc.
                CD-Plus/Extra
                session #2 starts at track 21, LSN: 268465, ISO 9660 blocks: 283825
                ISO 9660: 283825 blocks, label `ELL2                            '
              </cd-info:fullReport>
            </cd-info:analysisReport>
          </cd-info:cd-info>
        </mets:xmlData>
      </mets:mdWrap>
    </mets:techMD>

A full example METS file (which also includes EbuCore audio metadata, DFXML metadata and Isolyzer output) can be found here:

https://gist.githubusercontent.com/bitsgalore/ef164e953ca52218930a4bc512bfe96c/raw/c225a00ef997c99f1898d1f80783f334d357d5c9/mets-cdinfo-dfxml.xml

Thanks again to Euan and Bertrand for your suggestions!

Answered: I am looking for a straightforward accession tool. Can people please advise?

Wed, 07 Feb 2018 11:38:45 +0000

I would recommend Data Accessioner. It provides a nice GUI for the FITS toolset, creates and validates chechsums while moving content from external media, and provides a PREMIS XML file output (with flexible output available through subbing in different XSLT files) - http://www.dataaccessioner.org/

What are the risks involved in a DIY solution like RODA?

Wed, 07 Feb 2018 11:38:21 +0000

I am curious to hear from anyone, particularly smaller institutions, that have implemented DIY solutions with Archivematica, RODA, etc. We have a non-trivial amount of data (300TB - 400TB) that would be cost prohibitive in vendor-based pure cloud scenario using S3 and Glacier.

It seems to me the most cost effective method is to have data on-site in duplicate and replicated to Amazon Glacier, perhaps using a commercial system for the added support. However, its seems (in my ambitious mind) possible to manage a system like RODA in-house and handle the replication ourselves, needing only to pull from Glacier if the fixity checking reports an error. Can someone disabuse me of my fanciful notions? Any horror stories or success stories? Am I right to assume that as long as the bags are stored in Glacier we could migrate those to another system later on if we found maintaining our own system too unwieldy? How horrifying is it to all of you to rely on Glacier as a fail safe option?

Answered: IP limiting/restrictions for providing in-house access to copyrighted/restricted material?

Wed, 13 Sep 2017 01:22:02 +0000

We do this at NYPL.

Here's an example of something with IP restrictions.
https://digitalcollections.nypl.org/items/66f1db18-ac47-4160-8b3f-c3e8a595815f

...and something public on the web

https://digitalcollections.nypl.org/items/c90cfcb0-f875-0130-a5d6-3c075448cc4b

...and something IP-restricted linked out of a finding aid

http://archives.nypl.org/scl/22026#c1279014

IP restrictions are limited by collection to the building that houses the collection. One of the big questions that we still have is making access available more generally outside of a collection's building. That's requiring high-level administration involvment.

Answered: Managing Timestamps

Mon, 22 May 2017 14:52:49 +0000

We use checksums to prove bit-level preservation; the NDSA report Checking Your Digital Content: What is Fixity, and When Should I be Checking It? has been a very useful guide to help our institution establish and improve our fixity-checking for preservation and authenticity.

Answered: Ingesting large & hybrid digital collections

Mon, 22 May 2017 14:45:34 +0000

My institution has some experience with larger, heterogeneous collections, though our local practice is still evolving as we add larger and more complex collections to our holdings. I only have one link I can send your way as an example -- a collection of digital files relating to a redistricting project: http://www2.mnhs.org/library/findaids/gr00558.xml. This finding aid dates back to 2012 and is already a bit outdated for what we plan to do with new/incoming collections, but I think it gives you the general idea.

As you can see from that example, we (generally) rely heavily on original order, and use our descriptive tools (generally EAD finding aids) to try and aid researchers in finding what they're looking for. When there are dependencies between multiple digital objects, we keep them all together in a single .zip download rather than separate them out. The level of description varies in our collections; we previously described every file (as in the above example), but with newer and more homogeneous collections (such as large collections of digital photographs) we tend to describe them at the series level.

I hope that provides some help!

Answered: Incomplete ISO image after imaging CD-ROM - how to prevent and detect this?

Mon, 16 Jan 2017 13:12:25 +0000

And here's another update; following some additional research I did into this I wrote a blog post:

Detecting broken ISO images: introducing Isolyzer

This also discusses the detection script which I linked to earlier; this has been given a major overhaul and now works for hybrid ISO 9660 / Apple images as well. See:

https://github.com/KBNLresearch/verifyISOSize

Answered: Documenting Scope and Content for Web Archives

Wed, 21 Dec 2016 16:43:16 +0000

Not sure this is quite what you are after, but we did some work documenting the scope and content of one of our collections here: http://data.webarchive.org.uk/opendata/ukwa.ds.2/#issues and here: http://data.webarchive.org.uk/opendata/ukwa.ds.2/host-linkage/#issues

Answered: Are there accepted alternatives to WAVE for storing audio files larger than 4GB?

Wed, 17 Aug 2016 15:57:20 +0000

We have preserved a few > 4GB WAVE files, we make sure they conform to the RF64 standard. PRONOM was updated recenlty with identification of RF64 WAVE files.

Answered: How can I set up a test instance of Archivematica on OS X (10.11.2)?

Fri, 24 Jun 2016 14:29:35 +0000

Here are instructions for installing Archivematica using Vagrant, for development or testing purposes:

https://wiki.archivematica.org/Getting_started#Ansible_.26_Vagrant

If you have trouble, feel free to jump into the Archivematica Technical group: https://groups.google.com/forum/#!forum/archivematica-tech

Cheers!

Answered: Best SIP / AIP creation practices for optical carriers that span multiple volumes

Wed, 22 Jun 2016 18:29:18 +0000

In our audio digitization projects, it's common to have a single intellectual object span multiple pieces of media; for example, a symphony recorded across the front and backs of a vinyl record. Each face of the record is captured to its own preservation master file. In addition, we create an edit master from each preservation master for accessibility reasons.

In pseudo-JSON it looks like this.

symphony1:{
record1: {
  face1:{record1_face1_pm.wav, record1_face1_em.wav},
  face2:{record1_face2_pm.wav, record1_face2_em.wav}
},
record2: {
  face1:{record2_face1_pm.wav, record2_face1_em.wav},
  face2:{record2_face2_pm.wav, record2_face2_em.wav}
}
}

In our digitization workflows we ask for the files from each piece of media to be stored in its own bag. In the example, we would receive 2 bags (record1 and record2) that we treat as SIPs. Our repository is Fedora based, so each file set (preservation master, edit master, other derivatives) is stored as one Fedora object, and larger intellectual objects are aggregated through additional Fedora objects. I'm still mapping how this structure fits the OAIS AIP and AIC concepts.

Answered: Is there a GUI alternative to Bagger that performs a similar function?

Tue, 03 May 2016 16:22:26 +0000

Adding another relevant tool for anyone who happens on to this question:

Exactly: https://www.avpreserve.com/tools/exactly/

Answered: DIY digitisation, good or bad?

Tue, 05 Apr 2016 13:41:22 +0000

As long as the imaging device you use is capable of creating the quality of scans you want, I don't see how changing the device you use to create the scan would be a problem.

There are some things to check and to consider in order to make sure that happens, of course. I would personally double-check that the optical resolution of the cheaper desktop scanners (not the interpolated resolution) is sufficient to meet your resolution requirements, and that the camera set-up will give you the kinds of images you're looking for (e.g., resolution, lighting, camera stand, etc.).

The bigger concern (in my opinion) is all of the administrative questions that need to be answered: Are you able to organize and preserve these new outputs in a way that is consistent with (or part of) your existing procedures? How will you decide what bypasses the existing procedures? Who will be responsible for this new scanning? What are the cost implications? How will you provide access to the files? Would it be better to improve the current workflow instead of adding a new one?

Hope that helps!

Answered: Setting a custom date for wget or wpull?

Thu, 17 Dec 2015 18:40:14 +0000

A useful analog may be our work on the SLAC web archive. In that case, we web-published then re-captured historical web content with wget into WARC, then adulterated the CDX indices to reflect the "true" timestamps corresponding to the dates of the backups from which we restored the web content. That way, we had both a faithful record of the actual HTTP communications in the WARCs and timestamp-appropriate indexing for the purposes of access.

Answered: Strategy for preserving scanned files outside of repository

Sun, 06 Dec 2015 20:26:56 +0000

Here's our final solution: putting the files into Bagger/BagIt bags, then storing in our Research Data Store.

We also set up a mechanism to verify all the checksums on a 2-monthly basis, and added (non-public) links to the bags from our catalogue and research repository.

Thanks, everyone, for your help :-)

Answered: How should an organisation QA the results of its outsourced web archiving activities?

Wed, 25 Nov 2015 19:45:43 +0000

Great question! For the Parliamentary Archives, we have done exactly as you describe in terms of outsourcing the majority of our web archive capability to the Internet Memory Foundation. The justification for doing so is simple; we simply wouldn’t have the resources or infrastructure to do so in-house. The Houses of Parliament web estate forms a vital corporate record of the organisations activity which should be preserved and made accessible.

To set our Quality Assurance practices into context, it will help to give a very brief introduction to our web archive process:

First, Archive staff sign-off the Seed List (usually around 30 URL’s).Second, IMF carries out the crawls. There is the option for Archives staff to set off crawls using ArchiveTheNet (AtN).
An initial QA stage is then done by IMF to identify and rectify initial issues (such as ensuring crawl parameters have been met, technical limitations noted etc.).
Using project management software (JIRA) IMF then hand over the captured URL’s for the second QA stage which is carried out by Archive staff. We assign one person to lead the QA and then distribute QA tasks across multiple staff.
Any further QA issues are reported through JIRA and resolved by IMF.
Finally, captured crawls are delivered by IMF to the Archives for ingest into the Parliament’s digital repository (Preservica Enterprise Edition).

As you can see, there are multiple levels of QA which occur. From the initial QA carried out by IMF, to the second QA undertaken by the Archives. I would argue that there is a final level of QA involved during the ingest workflow as the ARCs/(W)ARC’s are characterised, validated, and virus scanned etc. Setting aside the ingest workflow, the bulk of our QA is a manual human process which involves a significant amount of time checking the captured data. We also have a Service Level Agreement (SLA) with IMF which is vital in ensuring that the required service and QA approaches are met. For obvious reasons, I can’t go into too much detail in terms of what the SLA contains!

I am aware that our process could be improved, and that our QA steps rely upon manual intervention. I’d be happy to discuss in more detail the QA steps I have outlined and potential areas of improvement. I’d also be particularly interested to hear from other web archive experts with suggestions as to what we should be doing as part of our QA activities.

The Parliamentary Archives pages for the web archive can be found here: http://www.parliament.uk/business/publications/parliamentary-archives/web-archive/

The actual web archive portal (hosted by IMF) is here: http://webarchive.parliament.uk/

~Chris Fryer, Senior Digital Archivist, Parliamentary Archives

Answered: Best method to record track playing order for ripped audio CDs

Tue, 20 Oct 2015 18:52:58 +0000

At home, I like using Exact Audio Copy.

It can build a cue sheet which not only keeps the track playing order of a CD intact, but also does something about the gaps between tracks. For some people who do this kind of thing, the ideal would be to rip a segued CD like "Dark Side of the Moon" which exists as separate tracks, but plays back gapless. The cue sheet is apparently able to do this.

However, cue sheets are not supported everywhere, so probably useless for you and not so good for digital preservation. But thought I would mention it.

CD-ROM / DVD imaging: is it customary to save scans of booklets / covers as well?

Tue, 22 Sep 2015 12:48:07 +0000

I'm trying to compile a minimum set of metadata that an end-user would need for using ISO images from a ~15,000 CD-ROM /DVD collection. In particular I'm wondering about CD booklets and covers. They often contain lots of useful info and documentation related to technical environments and usage instructions. However, having to digitize them would add a substantial overhead on top of the actual imaging workflow. An other option might be to make scans of the actual CDs, which often contain useful info as well. This is what e.g. Internet Archive seems to be doing:

https://archive.org/details/SharewearBreakthroughUtil_ProdCol

I'm curious how other institutions are dealing with this?

What organizations are preserving software?

Thu, 13 Aug 2015 20:04:13 +0000

I've been building a list of organizations that are attempting to preserve software including:

NIST's software reference library
some game archives such as Stanford's
Microsoft has an internal Archive
the Internet Archive is doing amazing work
so is the National Library of Australia
the play it again project
the University of Michigan Archive
Rhizome are preserving software based/dependent art
as is MoMA
and of course the computer history museum is also

There are resources here and here with more information but I'm interested in community input.

But who else is doing software preservation?

Would anyone else find such a list useful?

Is it problematic to include spaces in file names?

Thu, 09 Jul 2015 15:53:37 +0000

I have read about and heard different opinions about including spaces in file names, and based on casual observation it seems like, as a community, we typically recommend that people do not use spaces in their file names (perhaps for many of the reasons outlined here). Do you have experience with working with spaces in file names that provides insight into whether they are problematic? Should we continue to recommend that digital object creators avoid spaces in file names, or is it an obsolete concern?

Favorite online resources to learn more about DP tools?

Tue, 07 Jul 2015 20:13:52 +0000

I'm trying to gather information about some ways for folks just starting out in digital preservation to learn more about tools and resources. Any one have favorites, or know of good webinars that can offer more insight on tools? Any information sessions (preferably free) that you'd like to share that you found helpful?

I'm aware of the resources through NDSA and DPOE, and often refer to the COPTR tool grid. But I'd love to see if anyone has other favorites or has a resource they found particularly helpful. For example, the new ArchivesDirect service hosts information sessions and can be a good place for people to learn more about the new hosted service.

Thanks!

What are best hardware/software for automated back-up to external hard drive?

Tue, 09 Jun 2015 21:39:20 +0000

I'm wondering if we as a digipres community have centered on one or a few trusted solutions for ongoing, automated backup of personal data to physical media, i.e., external hard drives. I'm interested in hardware and software recommendations for a Windows environment. This would be active research data management, and the data is potentially valuable for the long-term. Any data identified for long-term preservation would be migrated to a trusted repository. Please, if there's anyone out there who can help me hone in on some trusted solutions of the zillions of options that seem to be out there in the PC world, I'd be much appreciative.

Recommendations - Drives for Offline Storage

Thu, 04 Jun 2015 18:36:34 +0000

I'm looking into options for drives for 'cold' storage for my institution, possibly for long periods of time (we would like a shelf-life of about four years). They don't need to be particularly fast so much as reliable and long-lived (and, of course, cost-effective). We've already got some LTO tape we use; we're looking more for hard disk drives or solid state drives to be held offline in a climate-controlled area. The scenario involves a write-once, read-a-few-times situation.

So, first of all, does anybody have any experiences or know of any studies about the longevity and data integrity of HDD versus SSD? I know, for example, that SSD is better-equipped for unpowered storage than HDD, but that they can also be more prone to data loss.

Second, does anybody have a particular brand that they use and would particularly recommend to purchase or to avoid?

If you'd rather not answer on-list, you can email me at sarah.barsness@mnhs.org and I'll post your replies to this thread anonymously for you (I could *really* use some very candid recommendations).

Preserving PDF files with multimedia content

Wed, 20 May 2015 20:24:50 +0000

I am curious what other insitutions are doing for PDF files with embedded video or audio content? We migrate almost all of our PDF's to PDF/A. The PDF/A stadard does not allow for embedded content like this.

We have discussed normalizing the file to ensure long term usability but not to PDF/A or extracting the embedded content and handling them seperately. But this takes them out of their original presentation.

Are there other options?

Thanks.

How to distinguish 68k and PPC executables on classic Macintoshes?

Thu, 26 Mar 2015 18:48:45 +0000

When getting a binary for a classic Macintosh operating system (up until version 9), how to distinguish for which processor architecture the binary was made?

Scanned manuscript - saved as TIFF image files in archival disc. Is it Ok

Mon, 16 Feb 2015 19:58:42 +0000

Any LTP "Packaging Format" Standard you Support that Makes More Sense than WARC?

Fri, 13 Feb 2015 21:20:18 +0000

Here's my question: Do you prefer a packaging format other than WARC? What other justifications should I take into account? Could you share additional resources for my consideration ( in addition to those below?) And I'll elaborate:

We generally agree that digital permanent records should be kept in open and supported format (tools and communities). We even have a broad selection of formats to chose from (from TIFF/JPEG2000 through PDF/A to XML). What remains, in my view, is a similar broad agreement on the package format that house the above content and linked metadata. The target here is the format of the container, wrapper, or capsule.

There are many requirements that this container should meet, and by far, I like that the container be extendable whereby one can add more content, additional metadata, user contributed context, etc.

Sofar though, .zip container has more or less led the charge, though other formats such as .warc are emerging from studies and comparaisons and showing more promise, which prompts me to ask:

Do you prefer a packaging format other than WARC? What other justifications should I take into account? Could you share additional resources I could consider in addition to these below?

Recommendations/studies supporting WARC
- For web objects: http://www.getpocket.com/a/read/777166755
- For all objects: https://fedora.phaidra.univie.ac.at/fedora/get/o:293682/bdef:Content/get
Standard using ZIP as container/wrapper/capsule:
- Victorian Electronic Records Strategy / VERS Encapsulated Object (VEO) : http://prov.vic.gov.au/wp-content/uploads/2014/05/VERSStdRevisionProposal-v1-0.pdf
Standard-to-be considering SIRF or an unspecified format (needs proof of concept)
- Storage Industry Networking Association's SIRF (Self-Contained Information Retention Format ( http://www.snia.org/SIRF ; )

Answers, links and comments will be well received!

Tool(s) for extracting administrative metadata from WARC?

Thu, 29 Jan 2015 07:08:14 +0000

I'm researching best practices for administrative metadata--preservation metadata in particular--for web archives. So far I've found some very helpful rationales and schemas, all PREMIS-in-METS-based, but I haven't seen anything that directly explains how one gets from point A to point B. That is, I haven't seen any descriptions of the steps nor the tools used to actually extract this type of metadata (or as much as can reasonably be gleaned) from WARCs. Have you? Have you extracted administrative metadata from your WARCs and lived to tell the tale? I'd love to know what you used and what you thought of the process.

Optimal file sizes for access PDFs

Sat, 24 Jan 2015 20:41:55 +0000

We have digitised a collection of old books, pamphlets, etc, which we will be making available as PDFs and as images through BookReader. The original files were scanned as TIFFs and will not be available to users. The PDFs we originally created were high quality and large file sizes; some over 100 MB. Obviously, this is far too large to expect most users - which will be the public - to download. Is it better to split the larger files into smaller and retain the high quality? Or regenerate the PDFs into smaller, lower quality files? And in either case, what is the ideal maximum file size for cultural heritage-type PDFs available to the general public? Have not been able to find any recommended standards at all around this issue.

Splitting PDFs while preserving quality

Wed, 21 Jan 2015 19:18:52 +0000

For easier presentation, we need to break some very large PDFs into smaller files (we already have the originals saved in TIFF format). However, when we do this via Adobe, the total file sizes get halved, e.g. a 600MB PDF results in 6 PDFs that total 300MB in size. Does this denote loss of quality? How do I check? And what's the best way to break large PDFs into smaller files without that loss of quality?

I thought this info would be relatively easy to find on the Adobe web site or elsewhere, but no luck so far.

What preservation format for video to use when digitizing VHS tapes?

Mon, 05 Jan 2015 19:57:23 +0000