Wikimedia Visitor Log Analysis Report

Wikimedia Visitor Log Analysis Report - Google requests

Daily averages, based on sample period: 1 May 2010 - 31 May 2010

This analysis is based on a 1:1000 sampled server log (squids) ⇒ all counts x 1000.
See also: Requests by destination or by origin / Methods / Scripts / Skins / Crawlers / Op.Sys. / Browsers / Google

This report shows all requests to Wikimedia servers where a Google server of service was involved in any way, be it the GoogleBot crawler or FeedFetcher collector scripts that run on Google servers, or a user that follows a link from a Google Web or Google Desktop search results page, or from Google Maps or Google Earth etcetera.
Technically speaking three fields in the squid log records are checked for this: client ip address, referer header and user agent header. A request can originate from an ip address which has been registered by Google and/or it can carry a referer tag that tells us a user clicked a link on a Google results page and/or it can carry an agent string that mentions a Google application which can reasonably be assumed to be genuinely Google's. See bottom of page for further details.
In total Google was somehow involved in 43.8% of daily external page^* requests
Google referred to our sites, through its services including search, maps, and Google Earth, 113,131,000 page views per day, representing 39.0% of our external page requests.
Including all of its different search crawlers and services hosted on its servers, Google itself requested another 14,071,000 page pages per day, representing 4.8% of our external page requests.
* = mime type text/html only

In order of request volume

Requests originating from a Google ip address

Service Total Pages Images Other

GoogleBot 11,293 9,885 589 818

Wireless 3,147 3,122 24 2

Translate 726 726 1

Other 405 203 31 171

FeedFetcher 317 129 12 176

Web search 6 6 1 1

Toolbar 1 1

Desktop 1 1

Maps 1 1 1

KeywordTool 1 1

Total 15,894 14,071 656 1,167

Requests originating from elsewhere

Service Total Pages Images Other

Web search 113,960 110,744 2,334 881

GoogleBot? 1,930 1,787 7 137

Desktop 1,005 46 1 959

Toolbar 759 129 36 594

Image search 588 302 281 5

Earth 222 29 193 1

Other 120 38 10 71

Maps 93 46 45 1

Translate 42 3 31 7

Mail 14 2 12 1

Wireless 3 2 1 1

FeedFetcher 3 3 1

Total 118,739 113,131 2,949 2,657

Top level domains

.com 33,799 32,625 821 353

.de 8,362 8,179 155 28

.uk 6,268 6,113 100 55

.jp 5,679 5,559 100 19

.fr 5,100 4,987 91 22

.br 4,532 4,465 54 13

.it 4,173 4,101 57 14

.pl 3,818 3,753 53 12

.ca 3,670 3,575 62 33

.mx 3,565 3,508 45 12

.es 3,210 3,139 57 14

.au 2,627 2,562 45 20

.in 2,051 1,925 43 84

.co 1,832 1,806 17 9

.ar 1,392 1,365 20 7

.nl 1,344 1,311 27 6

.ru 1,256 1,155 98 3

.ve 1,117 1,100 11 5

.tr 1,028 994 27 7

.cl 1,010 988 17 5

.se 936 915 17 4

.be 917 898 15 4

.pe 911 895 11 6

.id 880 663 212 6

.ch 816 797 14 5

.at 780 761 15 4

.fi 666 657 7 2

.th 624 515 102 7

.pt 605 591 11 3

.hk 575 526 31 18

.hu 555 537 15 3

.ro 547 524 16 7

.ph 540 511 22 7

.my 522 443 74 5

.vn 482 411 69 3

.nz 458 443 9 7

.ua 456 448 6 2

.il 443 432 7 4

.dk 427 416 10 2

.sg 409 381 24 4

.no 363 355 6 2

.ec 359 353 4 2

.gr 355 344 8 3

.cz 352 340 10 2

.ie 332 324 4 4

.bg 301 292 6 3

.hr 271 262 7 2

.tw 270 261 6 3

.za 244 235 5 4

.sk 239 228 7 4

.lt 218 213 4 1

.pk 213 199 5 8

.gt 203 200 2 1

.do 189 186 2 1

.eg 177 167 6 4

.si 155 150 4 1

.sa 142 136 5 2

.kr 139 98 30 11

.uy 137 134 2 1

.cr 136 133 2 1

.ma 135 131 3 2

.ae 132 129 3 1

.sv 128 125 2 1

.bo 112 110 2 1

.lv 95 93 2 1

.ee 95 94 1 1

.pr 95 93 1 1

.ba 83 80 2 1

.hn 65 63 1 1

.py 63 62 1 1

.lk 52 49 1 1

.jo 49 47 2 1

.by 46 46 1 1

.rs 45 43 2 1

.ke 42 41 1 1

.ng 42 40 1 2

.kz 39 38 1 1

.ni 39 38 1 1

.md 38 38 1 1

.tt 37 36 1 1

.kw 34 33 1 1

.qa 34 33 1 1

.jm 33 32 1 1

.is 28 27 1 1

.lu 28 28 1 1

.mt 27 26 1 1

.ge 26 24 2 1

.lb 25 24 1 1

.bh 24 23 1 1

.mu 20 19 1 1

.om 19 18 1 1

.cn 19 18 1 1

.ps 18 17 1 1

.gh 18 17 1 1

.pa 17 16 1 1

.np 16 15 1 1

.az 15 14 1 1

.cu 15 15 1 1

.ly 13 11 1 1

.cm 13 13 1 1

.ug 11 11 1 1

.sn 11 11 1 1

.am 11 11 1 1

.et 11 11 1 1

.ci 10 10 1 1

.mn 10 10 1 1

.mv 9 8 1 1

.fj 9 8 1 1

.tz 9 8 1 1

.bs 9 8 1

.cat 9 9 1

.bn 9 8 1 1

.mz 8 8 1 1

.bd 7 6 1 1

.dz 6 6 1 1

.net 6 1 5 1

.org 6 2 4 1

.zw 6 6 1 1

.kh 5 4 1 1

.bz 5 5 1 1

.uz 5 5 1 1

.na 4 3 1 1

.kg 4 4 1

.bw 4 4 1

.mg 4 4 1 1

.gy 4 4 1

.rw 3 3 1

.dm 3 3 1 1

.af 3 2 1 1

.cd 3 3 1

.ag 3 3 1

.vi 3 3 1

.vc 3 3

.gp 3 3 1 1

.bf 2 2 1 1

.gi 2 2

.li 2 2 1

.ht 2 2 1 1

.zm 2 2 1 1

.mk 2 2 1 1

.biz 1 1 1

.ls 1 1 1

.la 1 1

.ga 1 1

.sb 1 1 1

.ai 1 1 1 1

.me 1 1 1

.vg 1 1

.cg 1 1

.as 1 1

.sc 1 1 1 1

.vu 1 1

.ml 1 1

.bj 1 1 1 1

.gm 1 1 1 1

.to 1 1

.tl 1 1

.ad 1 1

.tg 1 1 1

.sm 1 1 1 1

.dj 1 1 1

.gg 1 1 1

.ne 1 1

.ki 1 1 1 1

.gl 1 1

.nu 1 1

.st 1 1

.ck 1 1

.ao 1 1

.tm 1 1

.ms 1 1

.pn 1 1

.ir 1 1

.sl 1 1 1

.tv 1 1 1

.us 1 1 1

.bi 1 1 1

.tk 1 1

.nr 1 1

.mw 1 1

.eu 1 1

.ac 1 1

.je 1 1 1

.cc 1 1 1

.sh 1 1 1

.cf 1 1

.tj 1 1

.im 1 1

.fm 1 1 1

.ws 1 1

undefined 19,329 15,573 843 2,913

Total 134,629 127,206 3,599 3,814

In alphabetical order

Requests originating from a Google ip address

Service Total Pages Images Other

Wireless 3,147 3,122 24 2

Web search 6 6 1 1

Translate 726 726 1

Toolbar 1 1

Other 405 203 31 171

Maps 1 1 1

KeywordTool 1 1

GoogleBot 11,293 9,885 589 818

FeedFetcher 317 129 12 176

Desktop 1 1

Total 15,894 14,071 656 1,167

Requests originating from elsewhere

Service Total Pages Images Other

Wireless 3 2 1 1

Web search 113,960 110,744 2,334 881

Translate 42 3 31 7

Toolbar 759 129 36 594

Other 120 38 10 71

Maps 93 46 45 1

Mail 14 2 12 1

Image search 588 302 281 5

GoogleBot? 1,930 1,787 7 137

FeedFetcher 3 3 1

Earth 222 29 193 1

Desktop 1,005 46 1 959

Total 118,739 113,131 2,949 2,657

Top level domains

.ac 1 1

.ad 1 1

.ae 132 129 3 1

.af 3 2 1 1

.ag 3 3 1

.ai 1 1 1 1

.am 11 11 1 1

.ao 1 1

.ar 1,392 1,365 20 7

.as 1 1

.at 780 761 15 4

.au 2,627 2,562 45 20

.az 15 14 1 1

.ba 83 80 2 1

.bd 7 6 1 1

.be 917 898 15 4

.bf 2 2 1 1

.bg 301 292 6 3

.bh 24 23 1 1

.bi 1 1 1

.biz 1 1 1

.bj 1 1 1 1

.bn 9 8 1 1

.bo 112 110 2 1

.br 4,532 4,465 54 13

.bs 9 8 1

.bw 4 4 1

.by 46 46 1 1

.bz 5 5 1 1

.ca 3,670 3,575 62 33

.cat 9 9 1

.cc 1 1 1

.cd 3 3 1

.cf 1 1

.cg 1 1

.ch 816 797 14 5

.ci 10 10 1 1

.ck 1 1

.cl 1,010 988 17 5

.cm 13 13 1 1

.cn 19 18 1 1

.co 1,832 1,806 17 9

.com 33,799 32,625 821 353

.cr 136 133 2 1

.cu 15 15 1 1

.cz 352 340 10 2

.de 8,362 8,179 155 28

.dj 1 1 1

.dk 427 416 10 2

.dm 3 3 1 1

.do 189 186 2 1

.dz 6 6 1 1

.ec 359 353 4 2

.ee 95 94 1 1

.eg 177 167 6 4

.es 3,210 3,139 57 14

.et 11 11 1 1

.eu 1 1

.fi 666 657 7 2

.fj 9 8 1 1

.fm 1 1 1

.fr 5,100 4,987 91 22

.ga 1 1

.ge 26 24 2 1

.gg 1 1 1

.gh 18 17 1 1

.gi 2 2

.gl 1 1

.gm 1 1 1 1

.gp 3 3 1 1

.gr 355 344 8 3

.gt 203 200 2 1

.gy 4 4 1

.hk 575 526 31 18

.hn 65 63 1 1

.hr 271 262 7 2

.ht 2 2 1 1

.hu 555 537 15 3

.id 880 663 212 6

.ie 332 324 4 4

.il 443 432 7 4

.im 1 1

.in 2,051 1,925 43 84

.ir 1 1

.is 28 27 1 1

.it 4,173 4,101 57 14

.je 1 1 1

.jm 33 32 1 1

.jo 49 47 2 1

.jp 5,679 5,559 100 19

.ke 42 41 1 1

.kg 4 4 1

.kh 5 4 1 1

.ki 1 1 1 1

.kr 139 98 30 11

.kw 34 33 1 1

.kz 39 38 1 1

.la 1 1

.lb 25 24 1 1

.li 2 2 1

.lk 52 49 1 1

.ls 1 1 1

.lt 218 213 4 1

.lu 28 28 1 1

.lv 95 93 2 1

.ly 13 11 1 1

.ma 135 131 3 2

.md 38 38 1 1

.me 1 1 1

.mg 4 4 1 1

.mk 2 2 1 1

.ml 1 1

.mn 10 10 1 1

.ms 1 1

.mt 27 26 1 1

.mu 20 19 1 1

.mv 9 8 1 1

.mw 1 1

.mx 3,565 3,508 45 12

.my 522 443 74 5

.mz 8 8 1 1

.na 4 3 1 1

.ne 1 1

.net 6 1 5 1

.ng 42 40 1 2

.ni 39 38 1 1

.nl 1,344 1,311 27 6

.no 363 355 6 2

.np 16 15 1 1

.nr 1 1

.nu 1 1

.nz 458 443 9 7

.om 19 18 1 1

.org 6 2 4 1

.pa 17 16 1 1

.pe 911 895 11 6

.ph 540 511 22 7

.pk 213 199 5 8

.pl 3,818 3,753 53 12

.pn 1 1

.pr 95 93 1 1

.ps 18 17 1 1

.pt 605 591 11 3

.py 63 62 1 1

.qa 34 33 1 1

.ro 547 524 16 7

.rs 45 43 2 1

.ru 1,256 1,155 98 3

.rw 3 3 1

.sa 142 136 5 2

.sb 1 1 1

.sc 1 1 1 1

.se 936 915 17 4

.sg 409 381 24 4

.sh 1 1 1

.si 155 150 4 1

.sk 239 228 7 4

.sl 1 1 1

.sm 1 1 1 1

.sn 11 11 1 1

.st 1 1

.sv 128 125 2 1

.tg 1 1 1

.th 624 515 102 7

.tj 1 1

.tk 1 1

.tl 1 1

.tm 1 1

.to 1 1

.tr 1,028 994 27 7

.tt 37 36 1 1

.tv 1 1 1

.tw 270 261 6 3

.tz 9 8 1 1

.ua 456 448 6 2

.ug 11 11 1 1

.uk 6,268 6,113 100 55

.us 1 1 1

.uy 137 134 2 1

.uz 5 5 1 1

.vc 3 3

.ve 1,117 1,100 11 5

.vg 1 1

.vi 3 3 1

.vn 482 411 69 3

.vu 1 1

.ws 1 1

.za 244 235 5 4

.zm 2 2 1 1

.zw 6 6 1 1

undefined 19,329 15,573 843 2,913

Total 134,629 127,206 3,599 3,814

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges
Agents: as for genuine agent strings: too many crawlers indentify themselves as 'GoogleBot' to take this at face value. They are accepted as genuine Google crawler requests only when the ip address matches a known range (see above). Other records that mention GoogleBot are counted as GoogleBot? (question mark, as this may include partners, like DoCoMo). However when the agent string mentions Google Desktop or Google Earth this is always accepted
Service: the service name is based on the agent string (plus for GoogleBot check for ip address, see above), if this is inconclusive it is based on the referer string.
Here is detailed breakdown per service of indicators that pointed to Google (total ≥ 3)

Service Total Originating from
Google ip address Referer mentions
Google url Agent mentions
Google service

Desktop 1,005 - - Y
Earth 222 - - Y
FeedFetcher 3 - - Y
FeedFetcher 317 Y - Y
GoogleBot 11,293 Y - Y
GoogleBot? 1,926 - - Y
GoogleBot? 4 - Y Y
Image search 588 - Y -
Mail 14 - Y -
Maps 93 - Y -
Other 110 - - Y
Other 9 - Y -
Other 229 Y - -
Other 175 Y - Y
Toolbar 659 - - Y
Toolbar 65 - Y -
Toolbar 35 - Y Y
Translate 41 - Y -
Translate 526 Y - Y
Translate 200 Y Y Y
Web search 113,960 - Y -
Web search 6 Y Y -
Wireless 3 - - Y
Wireless 3,147 Y - Y

Top Level Domain 'undefined': requests with top level domain 'undefined' are nearly all requests from anonymous ip addresses (crawler and other services)
Note: averages below 1 are always rounded up to 1

Generated on Sat, Jul 3, 2010 18:26
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.