What is Robots.txt File in SEO
1. What is Robots.txt File:
Robots.txt is a special text format that is not HTML or any other type. It gives the webmaster more flexibility to index an area of your website in SE.It is used for fast indexing ans crawling.Robots.txt files are used primarily to manage crawler traffic to your site, and usually to keep files from Google, depending on the file type.Before you create or edit Robots.txt files, you must know the limit of this URL blocking method. Depending on your goals and situations, you might want to consider other mechanisms to ensure your URL cannot be found on the web.
2. How does Google Crawler Works?
Crawl ==> Index [save] ===> Serve
You can use Robots.txt files for web pages (HTML, PDF, or other non-media formats that Google can read), to manage traffic crawling if you think your server will be overwhelmed by Google’s crawler request, or to avoid crawling pages not important or similar on your site.
3. What is the role of the Robots.Txt File?
Allow/disallow crawling
Handle duplicate content
Block any specific SE crawler
Sitemap information
4. What are the Robots.txt Parameters?
You can use Robots.txt files to block resource files such as images, scripts, or unimportant style files, if you think that the pages that are loaded without resources will not be significantly affected by losses. However, if the absence of this resource makes pages more difficult for Google’s crawler to understand pages, do not block it, or Google will not do good jobs in analyzing pages that depend on these resources.
# = comment
User-agent: ==> For which SE crawler I am writing
Allow: ==> what is allowed to crawl
Disallow: ==> what is disallowed to crawl
[Null] or [empty] ==> Nothing
/ [slash] ==> Everything in that directory
* [sterik] ==> Anything [Any crawler]
/?page= ==> Dealing a specific type of URL
Sitemap: ==> Inform sitemap URL to SE Crawler
Important Note:
Allow: ==> Disallow: / ==> Disallowed everything
Allow: / ==> Disallow: ==> Allowed everything
5. How to make Robots File?
Sample # 1
# This is the robots.txt file for www.newsupdatelive.com
User-agent: Googlebot
Disallow:
Sitemap: https://www.newsupdatelive.com/sitemap.xml
Sample # 2
# This is the robots.txt file for https://www.newsupdatelive.com/
User-agent: Googlebot
Disallow:
User-agent: *
Disallow: /
Sitemap: https://www.newsupdatelive.com/sitemap.xml
Sample # 3
# This is the robots file for www.newsupdatelive.com
User-agent: *
Disallow: /?page=
Disallow: /?search=
Sitemap: https://www.newsupdatelive.com//sitemap.xml
6. Important Note:
Use Robots.txt files to manage crawling traffic, and also to prevent image, video, and audio files from appearing in Google search results. This will not prevent another page or user from link to image, video, or your audio file.
When using robots.txt files, you need to be careful. Because if corrected wrongly, all SEO results will flow
Must place in the top level directory [index or home directory] of the website.
File text name [robots.txt] is case sensitive [you cannot keep Robots.txt or ROBOTS.TXT or etc]
Do not keep this file in any disallowed directory like /wp-content/
Some user-agents may bypass your standard robots.txt file
robots.txt file is public so do not put any private link here like admin URL or etc
Reading Material:
https://developers.google.com/search/docs/advanced/robots/robots_txt
