This post is about dealing with crawlers namely web-bots orelse search engines, this is Part IV of V linked to previous post SEO starter.
#Make efective use of robots.txt :
>Restrict crawling where it's not needed with robots.txt, A "robots.txt" file tells search engines whether they can access and therefore crawl parts of your site.There are a handful of other ways to prevent content appearing in search results, such as adding "NOINDEX" to your robots meta tag, using .htaccess to password protect directories.
>Use more secure methods for sensitive content, and Avoid allowing URLs created as a result of proxy services to be crawled.
#Be aware of rel="nofollow" for links :
>Fight comment spam with "nofollow", Setting the value of the "rel" attribute of a link to "nofollow" will tell Google that certain links on
your site shouldn't be followed or pass your page's reputation to the pages linked to.
>Make sure you have solid measures in place to deal with comment spam!
>Linking to sites that Google considers spammy can affect the reputation of your own site.
Next is Part V which is Promotion and Analysis.
