How web archives work: crawlers, captures and gaps

Crawlers and captures

A crawler is software that requests a web page, stores a copy and follows links to other pages. The Internet Archive says much of its archived web data comes from its own crawls or from crawls by Alexa Internet ¹. It says its crawls tend to find sites that are well linked from other sites ¹.

Each stored copy is a capture of a page at one moment. The Internet Archive says every day hundreds of web crawls contribute to the captures in the Wayback Machine ². A new capture appears in the Wayback Machine 3 to 10 hours after the crawl, by the Archive's account ¹.

Users can ask for a single capture. The Wayback Machine's Save Page Now feature saves a specific page one time. The Internet Archive says it does not save multiple pages, directories or whole sites, and it does not add the URL to future crawls ¹.

Reading the date in an archived URL

The Wayback Machine puts the capture time inside the archived URL. The Internet Archive gives the example web.archive.org/web/20000229123340/http://www.yahoo.com/ and says it was crawled on 29 February 2000 at 12:33:40 ¹. The code reads as year, month, day, hour, minute and second ¹. Replacing the code with an asterisk lists all captures of a site ¹.

The Memento protocol, published as RFC 7089 in December 2013, gives archives a common way to state this date. A capture served under Memento carries a Memento-Datetime header with the archival datetime of that capture ³.

What archives miss

Archives copy what crawlers can reach. The Internet Archive says it does not capture pages that need a password, pages reached only by sending a form, pages on secure servers and pages blocked by robots exclusions ². It says some sites are excluded at the owner's direct request ².

• Sites the crawlers did not know about at the time of the crawl ¹.
• Orphan pages with no links pointing to them ¹.
• Pages that depend on JavaScript or server-side image maps ¹.
• Forms or JavaScript features that need the original host, which may stop working in the archive ².
• At the Library of Congress, form and database driven content, streaming media, password-protected sites and paywalled content, which its crawlers typically do not archive ⁴.

Who decides what is kept

The Internet Archive takes exclusion requests by email at info@archive.org. It asks for the URLs, the time period to exclude and the period of control over the site, and says it makes no guarantees about the outcome ¹. Its copyright policy says it may remove content or disable access at its discretion ².

National libraries work under law or permission. The UK regulations let a deposit library request an online work by means of a web harvester ⁵. The Library of Congress says it works in a permissions-based environment and must at minimum give notice to site owners, with US government sites exempt from the notice rule ⁴. The British Library says it has collected websites since 2005, at whole UK domain level since 2013 ⁶.

Who runs web archives

The International Internet Preservation Consortium (IIPC) lists national libraries, archives, universities and non-profit bodies as members. They include the Library of Congress, the British Library, the National Library of Australia, the National Diet Library of Japan, the National Library Board Singapore, Bibliotheca Alexandrina in Egypt and the Internet Archive ⁷.

Courts and patent offices rely on these records. The European Patent Office tells its examiners that the Internet Archive is the best known archiving service and that its incompleteness does not detract from the credibility of the data it does archive ⁸.