Use a Dedicated Web Front End for Crawling

Joel Oleson has posted an article on the subject of Crawling and it’s consequences. He also presents a solution to the possible problem of the Crawling process being a heavy one . . . to use a dedicated web front end server for Crawling.

In SPS 2003 we found that indexing could be a pretty heavy weight operation.  In a small/medium environment it’s not a big deal.  In an enterprise environment crawling hundreds of GB to TBs indexing can potentially cause more traffic than user traffic.

You can find this article here.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Blog at

Up ↑

%d bloggers like this: