Devtantra Blog

Tutorials about Android, Angular, ReactJS, PHP, MySQL and Web Development

Work with robot.txt

Your site content is very important for your Site . The site crawl take most important part of your site , robot.txt tells search engine how they should visit your site .If you want to block specific search engine & also directories. So by the help of robot.txt you can do the job easily .The robot.txt file are text file which are placed on your web-server . They must be placed on the root folder, as an example…      www.yourwebsite.com/robots.txt

robot.txt

If your website does not have a robot.txt file then this is what happens –
A robot comes to visit. It looks for the robot.txt file. It does not find it because it isn’t there. The robot then feels free to visit all your web pages and content because this is what it is programmed to do in this situation.

Create robot.txt like that 

User-agent: *
Disallow:

In disallow you place that folder or search engine name which you want to block. SEE THIS   And User-agent: * means apply on all robots

Note

  • Didn’t block any popular search engine for Ex. Google, MSN etc.
  • Use allow tag for your site .
0
0
Summary
Work with robot.txt
Article Name
Work with robot.txt
Description
Your site content is very important for your Site . The site crawl take most important part of your site , robot.txt tells search engine how they should visit your site .If you want to block specific search engine & also directories. So by the help of robot.txt you can do the job easily .The robot.txt file are text file which are placed on your web-server .
Author

1 thought on “Work with robot.txt

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: