What are search engine robots?
Search engine robots, crawlers or spiders are automated agents that search engines run to index numerous Internet web pages. these search engine agents visit websites, navigate websites and read data of web pages by following links or leads from one website to another.
the functionality of these robots or spiders are very limited. there won’t be any kind of magic practice to make your website get a top rank in search engine result pages with these robots. All that search engines robots do is reading and making a copy of infORMation of meta tags, a page title and textual content of web pages. Search engines index these later. To make your site get a good rank in search results, what’s needed is search engine optimization. Web crawlers won’t miss out your important content if your web site is optimized for search engines.
– Not indexing
By editing “robots.txt”, you CAn make robots not access certain pages or directories using robots.txt.
(’*’ refers to all agents)
(’/’ means to block the whole site)
– Things to consider
the robots.txt file is placed under the root directory of a domain and CAn be accessed by anyone. You CAn just type “http://www.mydomain.com/robots.txt” to look at a site’s robots file. BeCAuse of security concern, you may want to consider removing all links to the directories that you don’t want robots to crawl or people to check out, instead of putting them explicitly in robots.txt by using “disallow”. Robots or spiders won’t access unlinked pages. Removing links to those directories CAn be better than using “disallow” in robots.txt.