John Andrews is a Competitive Webmaster and Search Engine Optimization Consultant in Seattle, Washington. This is John Andrews blog on issues of interest to the SEO community and competitive webmasters. Want to know more?

johnon.com  Competitive Web & SEO

What is Google Hiding? 403 Forbidden: “your query looks similar to automated requests”

This week Google started blocking people from using Google’s search engine. If you try and use advanced queries, such as you might use when you are trying to understand why Google is not indexing certain pages, you get a block message and are prevented from using Google:

Google Error We’re sorry…but your query looks similar to automated requests from a computer virus or spyware application. To protect our users, we can’t process your request right now. We’ll restore your access as quickly as possible, so try again soon. In the meantime, if you suspect that your computer or network has been infected, you might want to run a virus checker or spyware remover to make sure that your systems are free of viruses and other spurious software. If you’re continually receiving this error, you may be able to resolve the problem by deleting your Google cookie and revisiting Google. For browser-specific instructions, please consult your browser’s online support center. If your entire network is affected, more information is available in the Google Web Search Help Center. We apologize for the inconvenience, and hope we’ll see you again on Google.

This happens to me every time I try and look past page 10 or so of a search result this week. If I clear the Google cookies, as suggested, nothing improves. This is very frustrating, and has caused at last two people I work with to start using Live.com for search and guess what.. they like it! One says it is easier to use than Google, and the other says her website is ranked in MSN and that is a good enough reason to use it for a while and besides, Live search is pretty good.

Here’s one reason you need to go deep into Google’s results: run a site query to see what Google has indexed from your site, and Google replies with “page 1 – 10 of about 10 million results”. But if you click on the tenth page, you don’t get the 101 or so results. Instead you get a page that says “page 21-29 of about 29 results” plus a notice that the other 9 million or so they originally reported are “substantially similar” to the 20 already displayed. So which is it Google.. you have indexed 10 million pages or just 29 pages indexed?

We find out by clicking deep into the results. Google says there is a 19th page of results, and offers a link to it. Click it, and see the truth: Google is only indexing 29 pages, and even though it may know of 9 million others (we can’t be sure), it isn’t displaying those to users for this query (contrary to what it says on the results page).

But we don’t find out this week. This week, if you click into those later pages that Google says exist and provides links to, you get a 403 Forbidden Error. Google’s explanation (above) sounds like it is trying to instill fear and doubt into the Internet.. you may have a virus or spyware… as if to say it’s not Google’s fault. What is Google hiding? If Google.com was one of my websites, I would be afraid that Google would penalize it for being junky, because my internal links don’t actually work, and are deceptive to users.

★★ Click to share this article:   Digg this     Create a del.icio.us Bookmark     Add to Newsvine

27 Responses to “What is Google Hiding? 403 Forbidden: “your query looks similar to automated requests””

  1. Rip Says:

    This started happening to me today as well, although I wasn’t looking at search results that far down. I was using the links button that comes with one of the SEO toolbars.

    I get a 403 page with a captcha to enter and continue. The last time I did this, the captcha worked and I got my results. Today, the captcha works but sends me off to the same 403 page but without any way to continue.

    It looks like Google are going to stop allowing third party tools to work with their API.

  2. anonymouse Says:

    I can confirm this has been happening for awhile at a place I consult for – they run a lot of search queries against google and, every now and then, everyone behind the firewall starts getting blocked with that 403 message.

    The solution that worked for them was to change their external facing IP address. Cool if you have an IP range, not so cool if you have a standard dynamic IP from Comcast or wherever.

  3. Brian Says:

    All is good for me, no google issues. Also, I really don’t like Live Search. Most of the time it is much harder to find the results I am interested in.

  4. Liz Says:

    I have seen this behaviour for a long time. Both with captcha and without. I don’t know if it has always been a 403, though.

    @liz: Yes there has always been some handling of automated querying. That is not the point. These days Google is issuing that error for manual Google users, unrelated to automated querying.

  5. diarmuid ryan Says:

    i come across this quite often, mostly with third party serp checking software

    @diarmuid ryan: Yes, automated rank checkers should be given that message, as always. But this is not about automated rank checkers. This is human users doing manual queries of Google, getting that error message. What is Google hiding? Is the Mechanical Turk being used for manual-automated rank checking? Is Google Search Insights super-sensitive to repeated querying?  

  6. dstt Says:

    I’ve had this problem a few times, it first occurd to me a year ago. I don’t know which action exactly triggers it, but I can confirm that you are not the only one that has this problem ;).

  7. Chris Says:

    Anything to save a little disk space, right? I don’t think it’s the first time it’s happened though.

  8. Carsten Cumbrowski Says:

    Don’t worry Andrew, I got that message too once in a while and yes, when I did manual queries as a human, no scripts. You are not allowed to be too fast, because that means that you probably do not spend enough time looking at the nice Ads on the page and that makes Google angry :)

  9. Tim Says:

    I had this issue today, and did a search on msn.com (the next search engine that popped into my head) and found this blog about it (I’d usually prefer Yahoo next, ironically).

    Anyway, I was searching for a few very general terms on google to see where my site is placed and received this error myself. I thought at first it was due to the search string I had used, because I know google will block certain terms to filter out exploit-bots for some commonly insecure scripts, but this wasn’t the case.

    The captcha verification image is completely unreadable 95% of the time on that page, I’ve found, due to how they tack the letters together and distort them, so I think the fact that the first two times I tried to verify I wasn’t a “bot” didn’t help the situation when both times failed.

    It doesn’t seem to matter if I type in the correct captcha code now (if I’m even presented with the option in the first place), as it takes me to the forbidden error page. Extremely unhelpful error. I use Linux and Seamonkey, it’s not possible for my system to have a virus, etc., but I knew that as soon as I saw their error message “excuse” of why I was blocked.

    I did find that if I open another browser (say, Firefox) on this same system with the same IP, I’m still blocked (of course), but I’m presented with the captcha image and if I type in the correct verification code, it’ll allow me to search normally. Same with another system on this same home office network (same IP), provided I verify the image code. So, it seems to clearly be a cookie issue, which absolutely defeats the purpose of blocking search bots or some other user agent that doesn’t even use cookies anyway (I’ve not checked to be absolutely sure, as it could be some session or IP user agent).

    This is likely a ploy to cover up how they index poorly and favor vested interests, and it’s a good way for them to save resources on their systems by refusing to process too many searches for the same IP in X-number of minutes or hours. It’s likely one or both of those reasons. It sucks, as I’m always respectful and considerate with my wording and construction of my site and don’t abuse key words or meta tags, etc., but that gets me no consideration in response with my indexing anyway, and now I’m blocked. Lame.

    Another lame thing about google, is it must be the only search engine I know of that will find a single page match for a phrase (for example, say you type “Service with the best support”), and it’ll list it fine, but if you type it in without the quotes, you can go through 1,000 matches (I think they max out at 700, so let’s say 700 matches) and your page isn’t listed anywhere. What search engine doesn’t favor exact matches first, let alone to not list them at all?

    And, as someone else said above, it claim to have 29 million matches for some phrase or key word(s), yet even after you click on the link to search for “omitted results” and it lists a thousand of them, you’ll get maybe 7 pages of results (if you search at 100 results per page) and that’s it. Obviously this makes it very easy for them to favor advertisers and people with vested interests and only allows but a few select companies (700, max, and many of those 700 are repeats (ironically) where only maybe 100 results are unique companies/sites) to be able to be found.

    I play fair, I don’t complain and I accept how things are — it’s clear that if I want to be found, even if I have a 100% completely unique term or phrase that I’ll have to pay to be seen on google — but some of their tactics are a bit low handed and suspicious, and to be blocked for maybe 20 or 40 searches in an hour’s time, is a little much. Google is just another search engine, they are nothing new, nothing fresh and nothing exciting, and many existed long before they did… and I’m honestly not sure why they are the default, other than they usually get better matching on your terms, but when you consider the above problems, policies and favoritism, it becomes less valuable and more annoying over time. I’m not interested in using google if they are just going to be greedy and basically just have a favorite (paying) list of a few thousand companies and ignore the rest of the sites and companies that don’t want to pay a premium just to be indexed.

  10. James Boyer New Jersey Says:

    Interesting, I was able to get to the 35th page of results for a search that is supposed to have returned about 280,000 results. Why is Google doing this??

  11. Ishwar Says:

    Hi. I’m a teenager in Singapore and I just got this problem with google too! I have no spyware or stuff like that on my computer. I have even run multiple anti-virus scans and i have even rebooted my computer! Still the problem persists. Some sites I’ve visted assure me that the problem will cease to exist in about 24 hours. Is this true? Guys, I really need google to work! I’m completely reliant on this search engine for my research!

  12. Lyn Says:

    First time it happened to me was yesterday, my brother wanted an apple iphone so I tried to search for it and maybe compare prices and then this warning came up. and then after 2 hours I was able to browse through google again. when I went to google again today same message popped up. Normally I don’t go beyond page 2 if I’m searching for something. I’ve been using google since I learned how to use the internet. I was so alarmed because it says there might be spyware or something and I just bought an item online with my credit card. I’m searching through yahoo now to be clarified of this issue but all the sites I stumbled into are webmaster territory and I can’t understand what they’re talking about. Some say it has to do with firefox (i’m using firefox and internet explorer and it’s the same, can’t access google), some say it’s the one searching at fault because he is spamming whatever it is he is spamming. Any explanation? Do I have a spyware on my computer?

  13. cassis Says:

    I have links on my pages to Google translations of the same pages in different languages. UNtil today they worked fine. Now they get this same crappy message and following their recommendations about clearing cookies makes absolutely no difference.

  14. mobiel internet Says:

    @ Cassis, your request is somewhat automated. Does Google support the feature that you are using? You can alternative try Babelfish, which does almost the same.

  15. Jason Says:

    Google I hope you are reading this you piece of shit. How in the hell you gonna block people you think are bots when your blocking software is a bot. I will continue to blackhat the hell out of you.

  16. Theresa Says:

    Have you ever noticed this happening when you search for an individual person? Someone who doesn’t have a lot of searching happening on their name per se, but might be controversial in the tech sector? (Do a google on Sue Gardner, director of Wikipedia).

    John replies: yes good call. I notice an automated lookup on her name¬† is redirected to a default Google search page, instead of a results set. Looks like she’s on Google’s list of “managed” search queries. Hmm…..

  17. frank burns Says:

    I have the problem, but thought is was some sort of spyware, because the links of the virus things they want you to download look strange.

  18. Moe Sweet Says:

    I faced this error when I tried to create a new gmail account for my dad.
    I am from Burma which is at war with the US. Can it be the reason?

  19. KingCast Says:

    Happened to me this morning and can’t even get in to my own damn blogs. Refreshed cookies and no avail.

  20. htc touch pro 2 Says:

    I experienced this too this morning, how anoying >

  21. iceman Says:

    I am getting this on Firefox and safari, after rebooting too and cleaning cookies in safari. my gmail still works, thank god. also after resetting safari.

    i have a mac and ran a virus scan….nothing.

    Yahoo, fine
    MSN fine
    ask.com fine

    google 403

    WTF

    bye google search

    maybe it is mad at me and blocked my ip address as I use the firefox add ons adblock plus and customize google for many years but I just put on WOT add on to firefox and that made it mad.

    who knows

    do you?

    thanks

  22. Jasmine Says:

    I’ve received this message for trying to open my documents. From what I’ve seen on the internet while trying to figure out how to solve the problem, it happens if you make too many queries on google. What frustrates me is that I have to use my documents daily from different computers, and this is the only convenient way I’ve found to do so, yet now google won’t let me access them. Does anyone know of a way to fix this?

  23. GaguiMarkJoyce Says:

    KILL svchost.exe …VIRUS…

    DOWNLOAD REANIMATOR!

    BEST SOLUTION!

  24. Dwight Stegall Says:

    When you receive Error 403 Forbidden simply delete the Google.com cookie in your browser’s options panel. Then sign in again to receive a fresh cookie. It’s that simple.

  25. diane Says:

    I installed the new firefox and within an hour i got this message and it has stayed with me for a few hours, i done a virus check but nothing, how long will it take for this message to go away

  26. Dave Says:

    Just because you scan for viruses and trojans and don’t find one doesn’t mean there isn’t one or more there. There probably isn’t a scanner made that finds them all and certainly not
    from inside the copy of the OS you are running to do the scan. There are things like
    Tripwire that come close but I doubt most of us are running something of that class.

    Analyzing the network traffic is the way to go but that pretty much requires a
    2nd computer….and even that isn’t a sure thing if it’s not doing whatever triggered
    Google while you are looking.

    -Dave

  27. Jeannie Says:

    I deleted search History and Cookies like someone suggested earlier. It did NOT work. I still get the ,403 Forbidden when I try to go to Bing to change my search. I have had nothing but MULTIPLE problems with Google Chrome!! HATE THEM. I am a student and this even happens when I try to go to my ONLINE CLASSES. Now I’m going to have to pay someone to fix this. If anyone finds a solution please email me. jeanniemcculley@yahoo.com
    Thanks