John Andrews is a Competitive Webmaster and Search Engine Optimization Consultant in Seattle, Washington. This is John Andrews blog on issues of interest to the SEO community and competitive webmasters. Want to know more?

johnon.com  Competitive Web & SEO

Google User Agent Strings and Cloaking

For all those “Black Hatters” out there cloaking Google by User Agent (heh heh) we have one new UA to add to your list. Michael Gray reports that Google is now using yet one more user agent string for it’s AdWords Quality Checker. So, if you are a cloaker and have been using UA strings like these:

Mediapartners-Google/2.1
Googlebot/2.1 (+http://www.googlebot.com/bot.html)
Googlebot/2.1 (+http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot/1.0 (googlebot@googlebot.com http://googlebot.com/)
Googlebot/2.0 beta (googlebot@googlebot.com)
Googlebot/2.0 (+http://googlebot.com/bot.html)
Googlebot/1.0 (googlebot@googlebot.com)
Googlebot/2.1w (+http://googlebot.com/bot.html)
Googlebot-w/2.1 (+http://googlebot.com/bot.html)
Googlebot/1.0
Googlebot-Image/1.0
Google WAP Proxy/1.0
Nokia-WAPToolkit/1.2 googlebot(at)googlebot.com

you now have one more to add:

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)

My my my, what has the world come to.

DISCLAIMER: Well, for some reason I feel I must save the newbie from getting misled by my tongue in cheek posts around here. First, it is always a bad idea to cloak based on User Agent. Anybody can spoof that from their own browser and reveal your deception. Also, Google can control UA as it wants, thus detecting your deception. The consequences of cloaking are said to be serious (I would not know :-)

It has been said for years that Google sends stealth bots out without UA strings, from non-Google IPs as a test for cloaking. I always believed it, but never tested it. Why should I? I know UA cloaking is easy to defeat, so why use it? Well now Mr. Gray shows pretty solid evidence that Google is more than willing to cloak for competitive gain. Proof. There ya go.

This post of mine jokingly suggests you add that generic IE UA string to your cloaking list… that would be stupid. It’s a basic UA string used by versions of Microsoft’s Internet Explorer. You’d be cloaking a large percentage of your users. Don’t do that, ok? Don’t be stupid. The point (if your head doesn’t already have one) is that you can’t effectively cloak by UA. See?

★★ Click to share this article:   Digg this     Create a del.icio.us Bookmark     Add to Newsvine

One Response to “Google User Agent Strings and Cloaking”

  1. SEO 网站优化推广 Says:

    Google蜘蛛有几种…

    Google的蜘蛛有两种,专注于搜索的Googlebot和专注于广告的Mediabot,两者抓取的网页内容会互相参考。…