If you are not getting traffic on your blog there may be an issue with robots.txt. Although in Blogger or Blogspot, it is already set by Go...
If you are not getting traffic on your blog there may be an issue with robots.txt. Although in Blogger or Blogspot, it is already set by Google as a default. You can also find an option to use a custom robots.txt file in the blogger blog. So you can use extra functionality according to your needs.
I have created here some basic things which are most important for a blogger blog. Here we have provided you a sample robots.txt file for your blogger or Blogspot blog, just change your domain name.
User-Agent: *
Allow: /?display=wide
Allow: /?m=0
Allow: /?m=1
Sitemap: https://www.pradeeptomar.com/atom.xml?redirect=false&start-index=1&max-results=500
Sitemap: https://www.pradeeptomar.com/atom.xml?redirect=false&start-index=501&max-results=500
Sitemap: https://www.pradeeptomar.com/atom.xml?redirect=false&start-index=1001&max-results=500
Sitemap: https://www.pradeeptomar.com/atom.xml?redirect=false&start-index=1501&max-results=500
Sitemap: https://www.pradeeptomar.com/atom.xml?redirect=false&start-index=2001&max-results=500
Sitemap: https://www.pradeeptomar.com/atom.xml?redirect=false&start-index=2501&max-results=500
Sitemap: https://www.pradeeptomar.com/atom.xml?redirect=false&start-index=3001&max-results=500
Sitemap: https://www.pradeeptomar.com/sitemap-pages.xml
If you want to remove some posts from search engines then you follow the given below tricks. The "Disallow: /" tells the robots that it should not visit the pages on the site.
Disallow: /search -
If you want to remove search pages from Google search then you can put this robot's custom tag in your file.
Disallow: /p/page-name.html -
If you want to remove all the posts from Google search then you can put this robots custom tag in your file.
Disallow: /yyyy/mm/post-name.html -
If you want to remove all the posts from Google search then you can put this robots custom tag in your file.
How to add robots.txt to Blogger?
To add Custom Robots. Txt to Blogger, Just follow these steps -
Sign in to your blogger blog.
Navigate to Settings >> Search Preferences ›› Crawlers and indexing ›› Custom robots.txt ›› Edit ›› Yes.
Now paste the above robots. txt file code in the box.
Click on the Save button.
and now, you are done!
My blog post indexed.but my blog not crawling past 20 days. Any solution?
ReplyDeleteHello Dhaval,
DeleteI was also facing same problem, But find solution
Just login to Google Console.
Remove your property and add again your website property using "URL Prefix".
check all the errors carefully within next 7 days.
I hope it will solve your problem.
Hello Dhaval, Please provide your blog url here.
DeleteAlthough you should also add this in your robot.txt
User-agent: Googlebot-Mobile
Allow: /