Distributed web crawling

Distributed web crawling is a distributed computing technique whereby Internet search engines employ many computers to index the Internet via web crawling. Such systems may allow for users to voluntarily offer their own computing and bandwidth resources towards crawling web pages. By spreading the load of these tasks across many computers, costs that would otherwise be spent on maintaining large computing clusters are avoided.

Published by

Gennaro Cuofano

Gennaro is the creator of FourWeekMBA which target is to reach over two million business students, executives, and aspiring entrepreneurs in 2020 alone | He is also Head of Business Development for a high-tech startup, which he helped grow at double-digit rate | Gennaro earned an International MBA with emphasis on Corporate Finance and Business Strategy | Visit The FourWeekMBA BizSchool | Or Get in touch with Gennaro here

Leave a Reply