Search Engine Robots and Spiders

What are search enne robots?
Search enne robots, crawlers or spiders are mated nts that search ennes run to index numerous Internet web ps. se search enne nts visit s, navigate s and read of web ps by following s or leads from one to anor.

funionality of se robots or spiders are very limited. re won’t be any kind of mac praice to make your get a top rank in search enne result ps with se robots. All that search ennes robots do is reading and making a copy of information of meta tags, a p title and textual content of web ps. Search ennes index se later. To make your site get a good rank in search results, what’s needed is search enne optimization. Web crawlers won’t miss out your content if your web site is optimized for search engines.

Controlling search enne spiders or robots
We can still control behaviors of search enne robots to a certain degree.

– Not indexing
By editing “robots.txt”, you can make robots not access certain ps or direories using robots.txt.

(’*’ refers to all nts)

(Note, this access manment can be done through meta tags of web ps. You can man individual ps by using meta tags.)

– Blocking bad crawlers
You can block harmful crawlers from indexing your .

(’/’ means to block whole site)

– Controlling multiple robots
You can control many robots at a time.

– Where should I put robots.txt file?
Place it in root direory of your .

– Things to consider
robots.txt file is placed under root direory of a domain and can be accessed by anyone. You can just type “http://www.mydomain.com/robots.txt” to look at a site’s robots file. Because of security concern, you may want to consider removing all s to direories that you don’t want robots to crawl or people to check out, instead of putting m explicitly in robots.txt by using “disallow”. Robots or spiders won’t access uned ps. Removing s to those direories can be better than using “disallow” in robots.txt.

– Useful s
List of Search Engine Robots, Spiders and Cralwers
How to write robots.txt

Source