Finding lost pages with the Wayback Machine CDX API (every capture under a URL)

The Wayback Machine's search box shows one URL at a time. The CDX API lists every capture it holds, which is how you find pages nobody linked to. The basic query https://web.archive.org/cdx/search/cdx?url=example.com/folder/&matchType=prefix&output=json&limit=200 Parameters that matter - url: the address, with or without http:// - matchType: exact (default), prefix (everything under that path), host (everything on that host), domain (the host and its subdomains) - from and to: timestamps as yyyyMMddhhmmss, and any prefix of that works, e.g. from=2003&to=2006 - filter: field:regex, e.g. filter=statuscode:200 to drop redirects and errors, or filter=mimetype:image/.* for images only - collapse=urlkey: one row per distinct URL instead of every capture of it - fl: choose fields, e.g. fl=timestamp,original,statuscode - output=json: easier to process than the default space-separated text The fields urlkey, timestamp, original, mimetype, statuscode, digest, length. Two rows with the same digest are byte-identical captures. Opening a capture https://web.archive.org/web/<timestamp>/<original> shows it with the Wayback toolbar and rewritten links. https://web.archive.org/web/<timestamp>id_/<original> returns the raw file as captured, which is what you want for images, video files and checking exact bytes. A worked search Looking for every page under a dead fan site's episode directory, captured between 2001 and 2005, distinct URLs only: https://web.archive.org/cdx/search/cdx?url=example.com/episodes/&matchType=prefix&from=2001&to=2005&filter=statuscode:200&collapse=urlkey&fl=timestamp,original&output=json Limits Large prefix queries are paged. If results stop, add showResumeKey=true and pass the returned resumeKey back. Be polite: do not hammer it in a loop. Documentation: https://github.com/internetarchive/wayback/tree/master/wayback-cdx-server When you find something, post the capture URL with its timestamp. A find without the timestamp cannot be checked.