• Register

Merging & Deduping WARC files

+1 vote
Is there any programme/script that will take a group of WARC files and merge them, removing exact duplicate responses ?

I realise this probably goes somewhat against good practice, but for reasons of space I would like to remove the approximately 90% replication of content (e.g. unchanged images) but retain the varying parts.
asked May 1 by richard (170 points)

Please log in or register to answer this question.