https://qanda.digipres.org/tag/deduplication Powered by Question2Answer https://qanda.digipres.org/1155/merging-%26-deduping-warc-files Is there any programme/script that will take a group of WARC files and merge them, removing exact duplicate responses ?<br /> <br /> I realise this probably goes somewhat against good practice, but for reasons of space I would like to remove the approximately 90% replication of content (e.g. unchanged images) but retain the varying parts. https://qanda.digipres.org/1155/merging-%26-deduping-warc-files Tue, 01 May 2018 12:06:57 +0000