How to add file Robots.txt for WordPress Website tutorial


how-to-add-robots-txt-for-wordpress

How to add file Robots.txt to WordPress Website - Creating a website is not enough. The main purpose that website owners want is to be listed at search engines displayed in SERP with certain keywords. This list of websites and the ability to display the latest content is mainly by search engine robots crawling and indexing the website. The webmaster controls how the robot analyzes the website by inserting instructions in a special file called robot.txt.


Here, I will show you how to install the robots.txt file WordPress Website for the best SEO website. Note that some WordPress websites do not need to be indexed by search engines.


What is a Robots.txt file?


Robot.txt is a text file located in the website's root directory that informs the search engine crawler not to crawl the website. It is also known as the Robot Exclusion Protocol that prevents search engines from indexing useless content.


In short, robot.txt tells search engine bots what they shouldn't crawl at your website.


This is how it works! When the search engine bot is about to crawl your website URL (that is, it crawls, retrieves information for indexing), it finds the robot.txt file on your side.


how-to-add-robots-txt-for-wordpress


Why create Robots.txt file for WordPress Website?


You usually do not need to add the robot.txt file to the WordPress website. Search engines index the WordPress website by default. However, for better SEO, you add the robot.txt file to the root directory to disagree with search engines accessing specific areas of your WordPress website.


How to create Robots.txt file for WordPress Website?


Login to WordPress web hosting control panel. Here, I am using Cloudways - Managed Cloud Hosting platform.


Go to the Server tab from the top menu bar and then get SSH / SFTP access from Server Manager → Main Information.


how-to-add-robots-txt-for-wordpress


Use any FTP server application to access WordPress database files. I am using FileZilla here. Launch it and then connect to your server using Master Credentials.


how-to-add-robots-txt-for-wordpress


Once connected, go to the WordPress database file folder / application. You will see different folders there.


how-to-add-robots-txt-cho-wordpress


Now go back to Cloud Platform and from the top left bar, go to Apps. Select the application you want to add the robot.txt file:


how-to-add-robots-txt-for-wordpress


From the left pane, go to Application Management → Application Settings → General. You will find the application directory name.


how-to-add-robots-txt-for-wordpress


Return to FileZilla and navigate to / application / (NAME) / public_html. Create a new text file here and name it robot.txt.


how-to-add-robots-txt-for-wordpress


Right-click the robot.txt file, and then click View / Edit to open the file in the text editor (Notepad is a handy option).


how-to-add-robots-txt-for-wordpress


See more: Speed ​​up your WordPress site


Advanced robots.txt for WordPress


Search engines like Google or Bing support the use of wildcards in robot.txt files. This wildcard is used to agree / disagree specific types of files across WordPress websites.


The asterisk (*) is used to handle a range of options / options.


how-to-add-robots-txt-for-wordpress


Here, * * means that every image starting with an online image, with the jpg extension, will not be indexed by search engines. A few examples of WordPress robots.txt.


For example, image1.jpg, image2.jpg, imagexyz.jpg will not be indexed by search engines.


The power of word * is not limited to images. You do not even agree to all files with specific extensions.


how-to-add-robots-txt-for-wordpress


The above statement will require all search engines to disagree with the entire file with the Indian pdf pdf & Indian file found in the download directory.


You don't even agree to the core WordPress directory using *.


how-to-add-robots-txt-for-wordpress


The above line requires the search appliance not to crawl directories starting with wp-.


For example: wp-include, wp-content, etc. will not be indexed by search engines.


The other wildcard symbol used in WordPress.txt robot file is the dollar symbol ($).


how-to-add-robots-txt-for-wordpress


The above command requires the search appliance not to index .php referrals, including .php referrals? Id = 123, etc.


But what if you just want to block referrals.php? You must only include the $ symbol immediately after the .php introduction file.


The $ symbol ensures that only .php referrals are introduced, not .php intros? Id = 123.


You can also use $ for the directory.


This instructs the search appliance to disagree with the wp content directory plus any directories placed inside wp content. If you just disagree with the wp content instead of all subdirectories, you should use the $ symbol.


The $ symbol ensures only wp content is not allowed. All folders in this directory are still accessible.


Below is the robots.txt file for the Cloudways website.


User-agent: *


Disallow: / admin /


Disallow: / admin / *? *


Disallow: / admin / *?


Disallow: / website / *? *


Disallow: / website / *?


The first line represents the User-agent. This refers to the search engine that is allowed to access and index the website. The full list of search engine bots is available here.


Where * means all search engines. You are assigned each search engine separately.


This will disagree with search engines crawling directory administrators online. It is usually not necessary with search engines to index this directory.


If your WordPress site is a website, then it is best to restrict search engine bots to not gather search queries.


If your website owns a sitemap. Adding URLs helps search engine bots find a sitemap file. This results in faster page indexing.


how-to-add-robots-txt-for-wordpress


What is included in Robots.txt for WordPress?


You decide which WordPress website you want included in the SERP. Everyone has their own views about installing robots.txt WordPress file. Someone advised against adding robots.txt file in WordPress. In my opinion, one should add then disagree / wp-admin / directory. The Robots.txt file is public. You can find robot.txt file on any website by visiting www.example.com/robots.txt.


Conclude


As you can see, the robot.txt file is an interesting tool for SEO. It agrees to show search engine robots what to index and what not to index. But it must be handled with care. Bad configuration leads to the destruction of the entire website. So be careful!


I finished with the robot.txt file at WordPress. If you have questions about placing a robot.txt file, please ask in the comments.