I saw a dialogue on Twitter this morning about the premise of having a characteristic in Robots.txt that could well well block every crawling AND indexing. It started with this tweet by Christian Thurston (@ct_oz):
“Hello John [Mueller], has Google knowing to be making it so as that the robots.txt file doesn’t appropriate block crawling, but additionally blocks indexation? To quote @willcritchlow: “I will’t see many situations where I are looking out for to block crawling but don’t are looking out for to block indexing”.
“That would be a first-rate alternate in expectations (and yes, we stay judge these objects regardless). End you can absorb some examples where this would trigger a visual enhance in search?” Mueller responded. “I’d admire to use away from including extra directives. I’m aloof no longer responsive to well-liked points precipitated by this documented performance … SEOs difficulty about indexing, but normally these URLs most animated unsightly for station:-queries (or if there isn’t other, better content on the positioning), so it feels synthetic?”
With over 20 years of ride in web page positioning, “I in fact absorb by no capacity encountered a station where a author desired to absorb a page indexed that they block for crawling in robots.txt. No longer even once absorb I considered that,” I tweeted in response. “It’s well-liked practice for me to educate folk who they’ve a more than a few: (1) block crawling, or: (2) prevent indexation, when what they are looking out for to stay is every. Level to: for trip extra of a converse for bigger sites where saunter budget is a converse.”
Will Critchlow of Search Pilot agreed, “100% agree. I will see the conservatism of no longer changing a protracted-standing well-liked, but I in fact absorb by no capacity considered, and could well’t think of a station where you’d are looking out for to block crawling but allow indexation.”
Nonetheless what stay YOU think? Would or no longer it is generous to absorb a characteristic in Robots.txt that allowed you to specify that pages you don’t are looking out for to absorb indexed? Purchase our speedily three-inquire of ballotbelow and let us know what you imagine.
Please keep in mind that for those who vote yes for doing this that it is probably you’ll well well wish to select up ALL the danger that at some level your dev employees could well well misapply the aptitude and NoIndex pages that you don’t need them to.
Opinions expressed in this text are those of the visitor author and no longer necessarily Search Engine Land. Workers authors are listed right here.
Contemporary on Search Engine Land
About The Author
Eric Enge is Total Supervisor of Perficient Digital, a fleshy-carrier, award-winning digital agency. Previously Eric changed into the founder and CEO of Stone Temple, additionally an award-winning digital advertising and marketing agency, which changed into obtained by Perficient in July 2018. He is the lead co-author of The Art of web page positioning, a 900+ page book that’s known in the industry as “the bible of web page positioning.” In 2016, Enge changed into awarded Search Engine Land’s Landy Award for Search Marketer of the Yr, and US Search Awards Search Persona of the Yr. He is a prolific author, researcher, teacher and a sought-after keynote speaker and panelist at main industry conferences.