Crawling a billion web pages in just over 24 hours (via gerikson) — discussion

#performance #web