Talk:Distributed web crawling

Computing Low‑importance

	This article is within the scope of WikiProject Computing, a collaborative effort to improve the coverage of computers, computing, and information technology on Wikipedia. If you would like to participate, please visit the project page, where you can join the discussion and see a list of open tasks.ComputingWikipedia:WikiProject ComputingTemplate:WikiProject ComputingComputing articles
Low	This article has been rated as Low-importance on the project's importance scale.

Internet Low‑importance

	Internet portal This article is within the scope of WikiProject Internet, a collaborative effort to improve the coverage of the Internet on Wikipedia. If you would like to participate, please visit the project page, where you can join the discussion and see a list of open tasks.InternetWikipedia:WikiProject InternetTemplate:WikiProject InternetInternet articles
Low	This article has been rated as Low-importance on the project's importance scale.

From Amillar, May 30, 2004:

The following is a proposed solution, but does Grub (or others) actually use this algorithm? in reference to:

One solution to this problem is using every computer connected to the Internet to crawl some Internet adresses (URLs) in the background. After downloading the pages, the new pages are compressed and sent back together with a status flag (changed, new, down, redirected) to the powerful central servers. The servers manage a large database and send out new URLs to be tested to all clients.

Unite both sections into one![edit]

I agree to join the subsection Parallelization Policy from the Web Crawler article into this Distributed Web Crawling article.

"It has been suggested that the section Parallelization policy from the article Web crawler be merged into this article or section."

Zoe, please do this for ease of reading and coherence.

relation to "Distributed Search Engine"[edit]

Distributed search redirects to this page, but it's often not what people need, they may well be looking for Distributed search engine. Should there be cross-references, or a disambiguation page? --Avirr (talk) 16:49, 2 February 2011 (UTC)[reply]

Is Grub dead?[edit]

The implementation section talks about Grub and Looksmart, in the current tense. However, the relation to Looksmart is in the past tense. Additionally, I think Grub may even be a dead project. Docmphd (talk) 21:22, 26 January 2012 (UTC)[reply]