.Google's John Mueller answered a question concerning why Google.com indexes web pages that are actually prohibited coming from crawling by robots.txt as well as why the it is actually secure to dismiss the related Browse Console documents regarding those crawls.Crawler Website Traffic To Inquiry Specification URLs.The individual asking the concern chronicled that robots were actually producing links to non-existent inquiry guideline Links (? q= xyz) to web pages along with noindex meta tags that are also blocked in robots.txt. What motivated the question is actually that Google.com is actually crawling the links to those web pages, acquiring obstructed through robots.txt (without envisioning a noindex robots meta tag) after that getting reported in Google Search Console as "Indexed, though blocked through robots.txt.".The individual talked to the adhering to inquiry:." However listed here is actually the significant question: why would Google mark pages when they can't also view the web content? What's the benefit in that?".Google.com's John Mueller confirmed that if they can not creep the webpage they can't observe the noindex meta tag. He likewise produces an exciting acknowledgment of the website: hunt operator, advising to neglect the results considering that the "common" consumers won't see those end results.He composed:." Yes, you are actually appropriate: if we can't crawl the web page, our team can not observe the noindex. That mentioned, if our company can't creep the web pages, after that there is actually certainly not a lot for our company to mark. Thus while you may observe several of those webpages with a targeted web site:- question, the typical individual will not view all of them, so I wouldn't bother it. Noindex is additionally alright (without robots.txt disallow), it merely suggests the URLs will wind up being crawled (as well as wind up in the Browse Console document for crawled/not listed-- neither of these standings induce issues to the remainder of the site). The important part is actually that you don't make them crawlable + indexable.".Takeaways:.1. Mueller's answer validates the limitations in operation the Internet site: search advanced hunt driver for diagnostic main reasons. Among those factors is actually due to the fact that it is actually certainly not linked to the routine search index, it's a different thing altogether.Google.com's John Mueller talked about the web site hunt operator in 2021:." The brief response is that a website: query is certainly not implied to be full, nor utilized for diagnostics objectives.A website concern is a specific kind of search that restricts the results to a certain web site. It is actually generally merely words web site, a digestive tract, and after that the website's domain.This concern confines the outcomes to a details internet site. It is actually not suggested to become a complete selection of all the web pages coming from that web site.".2. Noindex tag without utilizing a robots.txt is actually alright for these sort of circumstances where a robot is actually linking to non-existent webpages that are acquiring discovered by Googlebot.3. URLs with the noindex tag will definitely produce a "crawled/not recorded" entry in Look Console and that those won't possess a damaging result on the rest of the site.Go through the concern and address on LinkedIn:.Why will Google.com mark webpages when they can't also see the information?Featured Picture by Shutterstock/Krakenimages. com.