Robots.txt is a text file that notifies search engine spiders that specific pages or sections of a website should not be crawled. Robots.txt requests are recognised and executed by the majority of major search engines, including Google, Bing, and Yahoo. In this article, we will dive into what robots.txt is and how to do robots.txt for a WordPress website.
WHAT IS ROBOTS.TXT?
Search engines scan websites by following links from one site to the next, eventually crawling billions of links and webpages. This crawling behavior is sometimes known as “spidering.”
The search crawler will seek for a robots.txt file after arriving at a website, but before spidering it. If one is found, the crawler will read it first before moving on to the next page. The information obtained in the robots.txt file will instruct additional crawler behaviour on this particular site because it contains information about how the search engine should crawl. If the robots.txt file does not contain any directives that disallow a user-agent from crawling other information on the site (or if the site does not have a robots.txt file), it will crawl other information on the site.
In simple words, it is a method of asking the crawlers not to crawl the particular page.
HOW TO DO IT FOR A WORDPRESS WEBSITE?
- First install the “All in one SEO pack” plugin.
- Then in your WordPress dashboard, go to the “Pages” section.
- And select the page that you don’t want the crawler to crawl.
- Open the page and scroll down where you find the title and description.
- Beneath that is an option named “No Index”.
- Select that checkbox and, in the top-right corner, press on “Update”.
- Once you do it, that particular page will never be indexed.
- Then for the server, go to robots.txt tester google – disallow : / (page name) – paste it in a notepad – save as robots.txt and submit it to the root folder.
- This is the easiest way to do it.
WRAPPING UP:
In this article, we have the easiest way for requesting the crawlers for not crawling or indexing a page for a WordPress website. We hope that this post answered all of your questions. If you like this article, be sure to check out our WordPress video lessons on YouTube.
