How should you scope a crawl for web archiving online discussion forums

Thu, 05 Jun 2014 13:35:47 +0000

A lot of popular online discussion platforms (phpbb, vbullitan, invision powerboard, etc.) generate a lot of different kinds of URLs for the same discussion threads and digital assets and do a lot of strange things with links for pagination and such. You can eaisly get stuck in a range of crawler traps. What are some good tactics to use when trying to scope crawling online discussion forms to archve them? So, going into scoping and planning to archive a discussion forum what kinds of ideas and tactics should one be thinking about/considering?

Is digitisation on topic?

Mon, 02 Dec 2013 13:24:06 +0000

There have already been a couple of questions focused on digitisation rather than digital preservation. I would assume that these are actually off topic and should be closed? The boundary is of course a little fuzzy.

I would propose that this community should be involved in decisions related to the results of a digitisation initiative. For example, the file formats and metadata schemas used, where/how the results are stored, and so on. However, questions focused on how to engage in digitisation, what types of scanners should be used or resolution to digitise at, will be off topic.

This question is about OCR, although is phrased in the title as a more general digitisation question. Is this in scope? Its a tough call.

As many of us in the digitisation community are well aware, confusing digital preservation for digitisation is a common mistake. I'd suggest adding some clear scoping detail on this to the "What kind of questions should I not ask here?" section of the FAQ. Examples may well be necessary to keep things clear.

Paul Wheatley

(from ZDPSE)

Digital Preservation Q&A - Recent questions tagged scope

How should you scope a crawl for web archiving online discussion forums

Is digitisation on topic?