SEO Expert Question: Download SEO Expert PDF
Explain significance of Robots.txt file in a website?
Answer:
Robots.text file is considered as a useful convention to prevent cooperating web robots and web crawlers from accessing all or part of a website or its content for which we don't want to be crawled and indexed but publicly view-able. It is also employed by search engines to archive and categorize website and to generate a rule of no follow regarding some particular areas of our websites.
| Previous Question | Next Question |
| What is XML sitemap for SEO Expert? | How to implemented keywords for optimization? |