Wikimedia Visitor Log Analysis Report

Wikimedia Visitor Log Analysis Report - Google requests

Daily averages, based on sample period: 1 Feb 2010 - 28 Feb 2010

This analysis is based on a 1:1000 sampled server log (squids) ⇒ all counts x 1000.
See also: Requests by destination or by origin / Methods / Scripts / Skins / Crawlers / Op.Sys. / Browsers / Google

This report shows all requests to Wikimedia servers where a Google server of service was involved in any way, be it the GoogleBot crawler or FeedFetcher collector scripts that run on Google servers, or a user that follows a link from a Google Web or Google Desktop search results page, or from Google Maps or Google Earth etcetera.
Technically speaking three fields in the squid log records are checked for this: client ip address, referer header and user agent header. A request can originate from an ip address which has been registered by Google and/or it can carry a referer tag that tells us a user clicked a link on a Google results page and/or it can carry an agent string that mentions a Google application which can reasonably be assumed to be genuinely Google's. See bottom of page for further details.
In total Google was somehow involved in 47.2% of daily external page^* requests
Google referred to our sites, through its services including search, maps, and Google Earth, 121,278,000 page views per day, representing 40.4% of our external page requests.
Including all of its different search crawlers and services hosted on its servers, Google itself requested another 20,565,000 page pages per day, representing 6.8% of our external page requests.
* = mime type text/html only

In order of request volume

Requests originating from a Google ip address

Service Total Pages Images Other

GoogleBot 18,832 16,710 718 1,403

Wireless 2,790 2,760 26 4

Translate 730 730 1

Other 432 263 43 127

FeedFetcher 277 92 6 179

Web search 9 9 1 1

Toolbar 1 1

Image search 1 1 1

Desktop 1 1

Earth 1 1

KeywordTool 1 1

Total 23,071 20,565 793 1,713

Requests originating from elsewhere

Service Total Pages Images Other

Web search 117,309 115,042 1,217 1,051

Other 5,228 1,140 2,187 1,900

Desktop 5,012 290 1 4,722

Image search 4,101 2,284 1,769 48

GoogleBot? 2,348 2,188 35 126

Toolbar 1,037 175 48 814

Earth 773 88 683 2

Maps 167 51 116 1

Translate 26 5 14 7

Wireless 14 13 1 1

Mail 9 2 5 1

FeedFetcher 1 1 1

Total 136,024 121,278 6,075 8,672

Top level domains

.com 41,768 39,849 1,196 723

.de 9,221 8,976 205 41

.uk 6,144 5,973 144 28

.fr 5,297 5,149 119 30

.jp 5,297 5,202 77 18

.mx 4,453 4,383 60 10

.it 4,433 4,339 75 19

.pl 4,107 4,031 58 18

.ca 3,878 3,769 97 11

.es 3,328 3,250 64 14

.br 3,106 3,053 44 9

.co 2,164 2,096 21 47

.in 2,059 1,998 50 11

.au 1,944 1,895 44 5

.nl 1,425 1,382 38 6

.ru 1,177 1,099 73 5

.tr 1,103 1,066 33 5

.se 1,045 1,016 24 5

.ph 1,007 968 32 8

.ve 937 921 12 4

.ar 889 868 17 4

.be 842 821 18 3

.at 825 797 23 5

.id 796 633 151 12

.fi 782 769 10 3

.th 750 660 85 5

.ch 744 725 17 3

.pt 602 587 13 3

.hu 589 561 22 6

.ro 570 547 19 4

.ua 463 454 7 3

.my 456 408 45 3

.pe 447 439 7 1

.cl 436 426 8 2

.il 429 419 7 3

.no 427 414 11 3

.dk 408 396 11 1

.sg 387 368 17 2

.gr 369 357 10 2

.bg 364 354 9 2

.ie 349 342 6 1

.vn 327 288 38 2

.nz 317 304 10 3

.cz 317 301 14 2

.hr 293 283 9 2

.gt 288 283 3 2

.cn 288 267 20 1

.za 248 241 6 1

.do 248 245 2 1

.sk 225 212 10 3

.ec 220 216 2 1

.lt 216 211 5 1

.tw 204 198 4 1

.pk 199 192 6 1

.hk 199 195 3 1

.sv 169 165 3 1

.si 160 155 4 1

.eg 149 144 5 1

.ma 146 143 3 1

.pr 144 142 2 1

.cr 122 119 2 1

.ae 121 118 3 1

.sa 113 108 4 1

.lv 112 108 3 1

.ee 104 102 2 1

.bo 96 94 1 1

.ba 91 87 3 1

.kr 90 75 15 1

.hn 86 84 1 1

.uy 68 66 1 1

.lk 53 52 2 1

.ke 49 48 1 1

.rs 47 45 1 1

.jo 45 43 2 1

.kz 43 42 1 1

.by 43 42 1 1

.md 43 42 1 1

.ng 42 41 1 1

.jm 41 41 1 1

.ni 40 40 1 1

.py 39 38 1 1

.tt 38 37 1 1

.bd 37 35 1 1

.is 33 32 1 1

.lu 32 31 1 1

.kw 31 30 1 1

.qa 29 29 1 1

.ge 26 24 2 1

.mt 26 25 1 1

.lb 24 24 1 1

.bh 19 19 1

.mu 19 19 1 1

.gh 18 18 1

.ps 17 16 1 1

.om 17 17 1 1

.az 15 14 1 1

.cu 15 14 1 1

.ly 13 12 1 1

.np 12 12 1 1

.sn 11 11 1 1

.am 11 11 1 1

.ci 10 9 1 1

.cat 10 9 1 1

.bn 10 9 1 1

.mv 9 8 1

.dz 9 9 1 1

.bs 9 9 1

.pa 8 8 1 1

.mn 8 8 1

.et 8 8 1 1

.fj 7 6 1

.tz 7 7 1

.bw 7 6 1 1

.bz 6 5 1

.net 6 1 5 1

.org 6 1 5 1

.mg 5 5 1 1

.mz 5 5 1 1

.zw 5 5 1

.dm 4 4

.ag 4 4 1 1

.kh 4 4 1

.uz 4 4 1

.gp 4 4 1

.rw 3 3

.na 3 3 1

.kg 3 3 1 1

.cd 3 2 1 1

.vi 3 3 1

.gy 3 3 1

.zm 3 3 1 1

.vc 3 3 1 1

.af 2 2 1 1

.ga 2 2 1

.me 2 1 1

.li 2 2 1

.ml 2 1 1

.ht 2 2 1

.mk 2 2

.ug 1 1

.gl 1 1 1 1

.ls 1 1 1 1

.la 1 1

.gi 1 1 1

.ai 1 1

.vg 1 1 1

.cg 1 1

.us 1 1 1

.as 1 1

.sc 1 1

.bj 1 1

.gm 1 1

.to 1 1

.ad 1 1 1 1

.tg 1 1

.sm 1 1 1

.dj 1 1 1

.gg 1 1 1

.ki 1 1 1

.biz 1 1

.nu 1 1

.cy 1 1

.st 1 1

.ck 1 1

.bf 1 1

.ao 1 1

.tm 1 1

.ms 1 1

.sb 1 1

.ir 1 1

.td 1 1

.sl 1 1 1

.tv 1 1 1

.bi 1 1

.tk 1 1

.nr 1 1

.vu 1 1

.gf 1 1

.edu 1 1 1

.nf 1 1

.mw 1 1

.eu 1 1 1

.ac 1 1 1

.gov 1 1

.je 1 1

.cc 1 1

.sh 1 1

.tl 1 1

.im 1 1 1

.fm 1 1 1

.ws 1 1 1

undefined 36,769 23,842 3,665 9,262

Total 159,085 141,841 6,858 10,379

In alphabetical order

Requests originating from a Google ip address

Service Total Pages Images Other

Wireless 2,790 2,760 26 4

Web search 9 9 1 1

Translate 730 730 1

Toolbar 1 1

Other 432 263 43 127

KeywordTool 1 1

Image search 1 1 1

GoogleBot 18,832 16,710 718 1,403

FeedFetcher 277 92 6 179

Earth 1 1

Desktop 1 1

Total 23,071 20,565 793 1,713

Requests originating from elsewhere

Service Total Pages Images Other

Wireless 14 13 1 1

Web search 117,309 115,042 1,217 1,051

Translate 26 5 14 7

Toolbar 1,037 175 48 814

Other 5,228 1,140 2,187 1,900

Maps 167 51 116 1

Mail 9 2 5 1

Image search 4,101 2,284 1,769 48

GoogleBot? 2,348 2,188 35 126

FeedFetcher 1 1 1

Earth 773 88 683 2

Desktop 5,012 290 1 4,722

Total 136,024 121,278 6,075 8,672

Top level domains

.ac 1 1 1

.ad 1 1 1 1

.ae 121 118 3 1

.af 2 2 1 1

.ag 4 4 1 1

.ai 1 1

.am 11 11 1 1

.ao 1 1

.ar 889 868 17 4

.as 1 1

.at 825 797 23 5

.au 1,944 1,895 44 5

.az 15 14 1 1

.ba 91 87 3 1

.bd 37 35 1 1

.be 842 821 18 3

.bf 1 1

.bg 364 354 9 2

.bh 19 19 1

.bi 1 1

.biz 1 1

.bj 1 1

.bn 10 9 1 1

.bo 96 94 1 1

.br 3,106 3,053 44 9

.bs 9 9 1

.bw 7 6 1 1

.by 43 42 1 1

.bz 6 5 1

.ca 3,878 3,769 97 11

.cat 10 9 1 1

.cc 1 1

.cd 3 2 1 1

.cg 1 1

.ch 744 725 17 3

.ci 10 9 1 1

.ck 1 1

.cl 436 426 8 2

.cn 288 267 20 1

.co 2,164 2,096 21 47

.com 41,768 39,849 1,196 723

.cr 122 119 2 1

.cu 15 14 1 1

.cy 1 1

.cz 317 301 14 2

.de 9,221 8,976 205 41

.dj 1 1 1

.dk 408 396 11 1

.dm 4 4

.do 248 245 2 1

.dz 9 9 1 1

.ec 220 216 2 1

.edu 1 1 1

.ee 104 102 2 1

.eg 149 144 5 1

.es 3,328 3,250 64 14

.et 8 8 1 1

.eu 1 1 1

.fi 782 769 10 3

.fj 7 6 1

.fm 1 1 1

.fr 5,297 5,149 119 30

.ga 2 2 1

.ge 26 24 2 1

.gf 1 1

.gg 1 1 1

.gh 18 18 1

.gi 1 1 1

.gl 1 1 1 1

.gm 1 1

.gov 1 1

.gp 4 4 1

.gr 369 357 10 2

.gt 288 283 3 2

.gy 3 3 1

.hk 199 195 3 1

.hn 86 84 1 1

.hr 293 283 9 2

.ht 2 2 1

.hu 589 561 22 6

.id 796 633 151 12

.ie 349 342 6 1

.il 429 419 7 3

.im 1 1 1

.in 2,059 1,998 50 11

.ir 1 1

.is 33 32 1 1

.it 4,433 4,339 75 19

.je 1 1

.jm 41 41 1 1

.jo 45 43 2 1

.jp 5,297 5,202 77 18

.ke 49 48 1 1

.kg 3 3 1 1

.kh 4 4 1

.ki 1 1 1

.kr 90 75 15 1

.kw 31 30 1 1

.kz 43 42 1 1

.la 1 1

.lb 24 24 1 1

.li 2 2 1

.lk 53 52 2 1

.ls 1 1 1 1

.lt 216 211 5 1

.lu 32 31 1 1

.lv 112 108 3 1

.ly 13 12 1 1

.ma 146 143 3 1

.md 43 42 1 1

.me 2 1 1

.mg 5 5 1 1

.mk 2 2

.ml 2 1 1

.mn 8 8 1

.ms 1 1

.mt 26 25 1 1

.mu 19 19 1 1

.mv 9 8 1

.mw 1 1

.mx 4,453 4,383 60 10

.my 456 408 45 3

.mz 5 5 1 1

.na 3 3 1

.net 6 1 5 1

.nf 1 1

.ng 42 41 1 1

.ni 40 40 1 1

.nl 1,425 1,382 38 6

.no 427 414 11 3

.np 12 12 1 1

.nr 1 1

.nu 1 1

.nz 317 304 10 3

.om 17 17 1 1

.org 6 1 5 1

.pa 8 8 1 1

.pe 447 439 7 1

.ph 1,007 968 32 8

.pk 199 192 6 1

.pl 4,107 4,031 58 18

.pr 144 142 2 1

.ps 17 16 1 1

.pt 602 587 13 3

.py 39 38 1 1

.qa 29 29 1 1

.ro 570 547 19 4

.rs 47 45 1 1

.ru 1,177 1,099 73 5

.rw 3 3

.sa 113 108 4 1

.sb 1 1

.sc 1 1

.se 1,045 1,016 24 5

.sg 387 368 17 2

.sh 1 1

.si 160 155 4 1

.sk 225 212 10 3

.sl 1 1 1

.sm 1 1 1

.sn 11 11 1 1

.st 1 1

.sv 169 165 3 1

.td 1 1

.tg 1 1

.th 750 660 85 5

.tk 1 1

.tl 1 1

.tm 1 1

.to 1 1

.tr 1,103 1,066 33 5

.tt 38 37 1 1

.tv 1 1 1

.tw 204 198 4 1

.tz 7 7 1

.ua 463 454 7 3

.ug 1 1

.uk 6,144 5,973 144 28

.us 1 1 1

.uy 68 66 1 1

.uz 4 4 1

.vc 3 3 1 1

.ve 937 921 12 4

.vg 1 1 1

.vi 3 3 1

.vn 327 288 38 2

.vu 1 1

.ws 1 1 1

.za 248 241 6 1

.zm 3 3 1 1

.zw 5 5 1

undefined 36,769 23,842 3,665 9,262

Total 159,085 141,841 6,858 10,379

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges
Agents: as for genuine agent strings: too many crawlers indentify themselves as 'GoogleBot' to take this at face value. They are accepted as genuine Google crawler requests only when the ip address matches a known range (see above). Other records that mention GoogleBot are counted as GoogleBot? (question mark, as this may include partners, like DoCoMo). However when the agent string mentions Google Desktop or Google Earth this is always accepted
Service: the service name is based on the agent string (plus for GoogleBot check for ip address, see above), if this is inconclusive it is based on the referer string.
Here is detailed breakdown per service of indicators that pointed to Google (total ≥ 3)

Service Total Originating from
Google ip address Referer mentions
Google url Agent mentions
Google service

Desktop 5,012 - - Y
Earth 773 - - Y
FeedFetcher 277 Y - Y
GoogleBot 18,832 Y - Y
GoogleBot? 2,347 - - Y
Image search 4,101 - Y -
Mail 9 - Y -
Maps 167 - Y -
Other 1,041 - - Y
Other 4,182 - Y -
Other 5 - Y Y
Other 134 Y - -
Other 297 Y - Y
Toolbar 897 - - Y
Toolbar 94 - Y -
Toolbar 46 - Y Y
Translate 25 - Y -
Translate 522 Y - Y
Translate 208 Y Y Y
Web search 117,309 - Y -
Web search 9 Y Y -
Wireless 14 - - Y
Wireless 2,789 Y - Y

Top Level Domain 'undefined': requests with top level domain 'undefined' are nearly all requests from anonymous ip addresses (crawler and other services)
Note: averages below 1 are always rounded up to 1

Generated on Thu, Mar 11, 2010 2:37
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.