Digital Preservation Q&A - Recent questions tagged warc

Best way to crawl website from localhost with wget, preserving all files in source directory

Tue, 03 Jul 2018 14:16:10 +0000

We recently recovered the contents of an old (2004) website from CD-ROM. I managed to get a local instance of the site running using the Apache web server; by editing the machine’s hosts file the site is available on that local machine from its original URL, which is http://www.nl-menu.nl (some more context can be found here).

I’m now looking into ways to crawl the contents of the site into a WARC, so we can ingest it into our web archive. After initial experiments with Heritrix failed, I moved on to wget. After some experimentation the following wget command appeared to work reasonably well:

Attempt 1: mirror from site root

wget --mirror \
    --page-requisites \
    --warc-file="nl-menu" \
    --warc-cdx \
    --output-file="nl-menu.log" \
    http://www.nl-menu.nl

However, closer inspection of the result showed that about 668 files in the source directory are missing in the resulting WARC file. The majority (90%) of these files are “orphan” resources that are not used/referenced by any of the HTML files in the crawl. However, the remaining 10% of missing files are resourced that are referenced, in most cases through JavaScript variables. These aren’t picked up by wget, and therefore they end up missing in the WARC. So I am looking for a way to force wget to include these resources anyway.

Attempt 2: use –input-file

At first wget’s --input-file switch (which takes a list of URLs) looked like a good way to achieve this. I created a directory listing of all files that are part of the website, and then transformed them into corresponding URLs:

find /var/www/www.nl-menu.nl -type f \
    | sed -e 's/\/var\/www\//http:\/\//g' > urls.txt

Then I ran wget like this (note that I removed the --mirror option, as this apparently causes wget to do a recursive crawl for each single URL in the list, which takes forever):

wget --page-requisites \
    --warc-file="nl-menu" \
    --warc-cdx \
    --output-file="nl-menu.log" \
    --input-file=urls.txt

This results in a WARC file that contains all files from the source directory: perfect! But it does introduce a different problem: when I try to access the WARC using pywb, it turns out that the WARC is made up of 85864 individual captures (i.e. each file appears to be treated as an individual capture)! This makes rendering of the WARC impossible (loading the list of capture alone takes forever to begin with).

Attempt 3: include list of URLs in crawl

So as a last resort I created a list of all URLs in HTML format, and put that file in the source directory. Steps:

Create list of URLS in Markdown format (add “<” and “>” pre-and suffix to each line):

find /var/www/www.nl-menu.nl -type f | sed -e 's/\/var\/www\//<http:\/\//; s/$/>\n/g' > urls.txt
Replace any whitespace characters with %20 to avoid malformed URLs:

sed -i 's/\ /%20/g' urls.txt
Convert URL list to HTML which is placed at the root directory of the source dir:

sudo pandoc -s urls.txt -o /var/www/www.nl-menu.nl/urls.html

Then I ran wget, using the above URL list as crawl root:

wget --mirror \
    --page-requisites \
    --warc-file="nl-menu" \
    --warc-cdx \
    --output-file="nl-menu.log" \
    http://www.nl-menu.nl/urls.html

The resulting WARC contains all files that are in the source dir, and it can be accessed as one single capture in pywb. The obvious downside of this hack is that it compromises the integrity of the ‘original’ website by adding one (huge) HTML file that was not part of the original site to the WARC.

This makes me wonder if there is another, more elegant way to do this that I have overlooked here? Any suggestions welcome!

BTW I know this question is somewhat similar to [this earlier one] (http://qanda.digipres.org/337/there-web-archiving-tool-that-produces-warc-directory-tree), but option 2 as mentioned by @anjackson there looks similar to Attempt 2 in my case.

Merging & Deduping WARC files

Tue, 01 May 2018 12:06:57 +0000

Is there any programme/script that will take a group of WARC files and merge them, removing exact duplicate responses ?

I realise this probably goes somewhat against good practice, but for reasons of space I would like to remove the approximately 90% replication of content (e.g. unchanged images) but retain the varying parts.

Setting a custom date for wget or wpull?

Thu, 17 Dec 2015 11:47:58 +0000

Both wget and wpull, when saving to WARC, will store the date and time of when the archiving action took place in every HTTP request.

Is there a way to modify this date, either during or after recording?

Any LTP "Packaging Format" Standard you Support that Makes More Sense than WARC?

Fri, 13 Feb 2015 21:20:18 +0000

Here's my question: Do you prefer a packaging format other than WARC? What other justifications should I take into account? Could you share additional resources for my consideration ( in addition to those below?) And I'll elaborate:

We generally agree that digital permanent records should be kept in open and supported format (tools and communities). We even have a broad selection of formats to chose from (from TIFF/JPEG2000 through PDF/A to XML). What remains, in my view, is a similar broad agreement on the package format that house the above content and linked metadata. The target here is the format of the container, wrapper, or capsule.

There are many requirements that this container should meet, and by far, I like that the container be extendable whereby one can add more content, additional metadata, user contributed context, etc.

Sofar though, .zip container has more or less led the charge, though other formats such as .warc are emerging from studies and comparaisons and showing more promise, which prompts me to ask:

Do you prefer a packaging format other than WARC? What other justifications should I take into account? Could you share additional resources I could consider in addition to these below?

Recommendations/studies supporting WARC
- For web objects: http://www.getpocket.com/a/read/777166755
- For all objects: https://fedora.phaidra.univie.ac.at/fedora/get/o:293682/bdef:Content/get
Standard using ZIP as container/wrapper/capsule:
- Victorian Electronic Records Strategy / VERS Encapsulated Object (VEO) : http://prov.vic.gov.au/wp-content/uploads/2014/05/VERSStdRevisionProposal-v1-0.pdf
Standard-to-be considering SIRF or an unspecified format (needs proof of concept)
- Storage Industry Networking Association's SIRF (Self-Contained Information Retention Format ( http://www.snia.org/SIRF ; )

Answers, links and comments will be well received!

Tool(s) for extracting administrative metadata from WARC?

Thu, 29 Jan 2015 07:08:14 +0000

I'm researching best practices for administrative metadata--preservation metadata in particular--for web archives. So far I've found some very helpful rationales and schemas, all PREMIS-in-METS-based, but I haven't seen anything that directly explains how one gets from point A to point B. That is, I haven't seen any descriptions of the steps nor the tools used to actually extract this type of metadata (or as much as can reasonably be gleaned) from WARCs. Have you? Have you extracted administrative metadata from your WARCs and lived to tell the tale? I'd love to know what you used and what you thought of the process.

Is there a web archiving tool that produces WARC + directory tree?

Tue, 09 Sep 2014 20:05:01 +0000

An important feature of any web archiving tool is the ability to specify that one would like to pull down embedded resources ("page requisites" in the parlance of wget) that are hosted on a domain other than the site that is the target of the crawl. Such cases are found not only as examples of people "hot-linking" other people's images, but is encountered heavily on sites with cloud based content delivery networks, such as Tumblr. Including such assets in a crawl is absolutely necessary when one's aim is to achieve a complete mirror of a site.

Heritrix has an option for including such assets in the crawl scope:https://webarchive.jira.com/wiki/display/Heritrix/unexpected+offsite+content

As does Httrack, using the --near flag:\

http://www.httrack.com/html/fcguide.html\ But of course, Httrack does not offer WARC output.

Wget has the -H flag, allowing one to "span hosts" (in other words hit sites with domain names other than the starting url), it lacks the ability to specify that the crawler should span hosts only for page requisites, and so tries to download the entire web if one combines an infinitely recursive crawl with -H. There are some hacky ways of getting around this, but they aren't pretty or reliable. The great thing about Wget though is that it allows the user to output WARC and a directory tree of the crawled site, thus not locking one in to WARC completely.

Are there any tools of the trade that allow one to conduct the comprehensive type of crawl mentioned above, but also have the affordance of outputting WARC, and a directory tree?