Devtantra Blog

Technical tuts & Thoughts

Tag: robot.txt

Work with robot.txt

Your site content is very important for your Site . The site crawl take most important part of your site , robot.txt tells search engine how they should visit your site .If you want to block specific search engine & also directories. So by the help of robot.txt you can do the job easily .The […]