Wikimedia Visitor Log Analysis Report

Wikimedia Visitor Log Analysis Report - Google requests

Daily averages, based on sample period: 1 Dec 2009 - 31 Dec 2009

This analysis is based on a 1:1000 sampled server log (squids) ⇒ all counts x 1000.
See also: Requests by destination or by origin / Methods / Scripts / Skins / Crawlers / Op.Sys. / Browsers / Google

This report shows all requests to Wikimedia servers where a Google server of service was involved in any way, be it the GoogleBot crawler or FeedFetcher collector scripts that run on Google servers, or a user that follows a link from a Google Web or Google Desktop search results page, or from Google Maps or Google Earth etcetera.
Technically speaking three fields in the squid log records are checked for this: client ip address, referer header and user agent header. A request can originate from an ip address which has been registered by Google and/or it can carry a referer tag that tells us a user clicked a link on a Google results page and/or it can carry an agent string that mentions a Google application which can reasonably be assumed to be genuinely Google's. See bottom of page for further details.
In total Google was somehow involved in 49.2% of daily external page^* requests
Google referred to our sites, through its services including search, maps, and Google Earth, 95,830,000 page views per day, representing 39.6% of our external page requests.
Including all of its different search crawlers and services hosted on its servers, Google itself requested another 23,323,000 page pages per day, representing 9.6% of our external page requests.
* = mime type text/html only

In order of request volume

Requests originating from a Google ip address

Service Total Pages Images Other

GoogleBot 23,269 20,309 433 2,527

Wireless 3,257 2,220 1,022 15

Translate 467 466 1

Other 315 225 31 60

FeedFetcher 275 95 1 180

Web search 9 8 1 1

Image search 1 1

Toolbar 1 1

Desktop 1 1

Maps 1 1 1

KeywordTool 1 1

Total 27,592 23,323 1,487 2,782

Requests originating from elsewhere

Service Total Pages Images Other

Web search 93,313 91,663 1,121 529

Desktop 5,873 339 1 5,534

Other 4,726 152 2,111 2,462

Image search 3,167 1,793 1,330 44

GoogleBot? 1,625 1,528 5 92

Toolbar 965 169 53 744

Earth 743 102 639 2

Maps 359 56 299 4

Mail 59 10 43 6

Translate 26 6 11 9

Wireless 12 12 1

FeedFetcher 1 1 1

Total 110,868 95,830 5,612 9,426

Top level domains

.com 34,206 32,703 1,222 282

.de 7,405 7,174 204 27

.uk 4,925 4,777 130 18

.fr 4,578 4,389 151 38

.jp 4,512 4,420 71 21

.pl 3,426 3,356 57 13

.it 3,349 3,268 69 11

.ca 3,148 3,049 85 14

.br 2,665 2,601 45 19

.es 2,448 2,381 57 10

.mx 2,259 2,208 43 8

.in 1,762 1,699 51 11

.tr 1,363 1,316 41 6

.au 1,326 1,280 36 9

.nl 1,208 1,167 38 3

.ru 1,093 1,020 68 5

.se 845 822 21 2

.ar 808 784 19 5

.be 709 688 19 2

.at 691 664 24 3

.fi 666 653 11 2

.ch 648 629 16 2

.ph 599 585 9 6

.pe 566 555 9 3

.th 560 543 13 3

.id 536 510 20 6

.cl 532 518 11 3

.co 501 481 9 12

.pt 491 478 11 2

.ve 482 473 6 3

.hu 473 448 20 6

.ro 432 412 17 2

.il 420 411 7 2

.ua 399 391 6 2

.vn 370 361 7 2

.cn 357 312 42 3

.my 349 338 9 2

.dk 333 321 11 1

.no 325 312 11 1

.sg 309 303 5 1

.gr 308 298 8 1

.bg 287 279 7 2

.ie 284 277 6 1

.cz 261 248 12 2

.nz 235 217 5 12

.hr 231 223 6 1

.tw 229 220 7 2

.ec 199 194 3 1

.eg 186 179 6 2

.hk 183 178 4 1

.lt 181 175 4 1

.pk 174 168 5 1

.sk 171 161 7 3

.ma 145 141 2 1

.si 142 137 4 1

.do 136 133 2 1

.za 125 122 3 1

.sa 119 114 4 1

.ae 113 110 3 1

.kr 97 79 18 1

.lv 91 88 3 1

.pr 90 88 2 1

.ee 84 81 2 1

.ba 78 74 3 1

.cr 70 69 1 1

.uy 61 59 2 1

.bo 51 49 1 1

.jo 50 48 2 1

.gt 48 46 2 1

.sv 46 45 1 1

.lk 43 41 1 1

.by 38 37 1 1

.rs 36 35 1 1

.md 35 34 1 1

.bd 34 32 1 1

.kz 32 31 1 1

.kw 32 31 1 1

.py 31 30 1 1

.hn 29 28 1 1

.ke 29 29 1 1

.ng 29 28 1 1

.jm 28 28 1 1

.lu 27 26 1 1

.tt 27 27 1 1

.qa 26 26 1 1

.is 25 24 1 1

.mt 25 24 1 1

.bh 23 23 1 1

.ge 20 18 2 1

.om 20 20 1 1

.lb 20 19 1 1

.ps 18 17 1 1

.ni 16 15 1 1

.mu 15 14 1 1

.ly 12 12 1 1

.gh 12 12 1 1

.az 12 11 1 1

.np 12 11 1 1

.cu 12 11 1 1

.et 12 11 1 1

.mn 10 9 1 1

.sn 9 9 1 1

.dz 8 8 1 1

.ci 7 7 1 1

.tz 7 6 1

.pa 7 7 1 1

.am 7 6 1 1

.bs 7 7 1

.cat 7 7 1 1

.bn 7 6 1 1

.mv 6 5 1 1

.kg 4 4 1 1

.kh 4 3 1

.gp 4 4 1 1

.dm 3 3 1

.fj 3 3 1 1

.bw 3 3 1 1

.mg 3 3 1

.mz 3 3

.bz 3 3 1 1

.uz 3 3 1 1

.net 3 1 3 1

.org 3 1 2 1

.zw 3 3 1 1

.ug 2 2

.rw 2 2 1 1

.af 2 2 1

.na 2 1 1

.li 2 2 1

.cd 2 2 1 1

.ag 2 2

.vi 2 2 1 1

.ht 2 2 1 1

.gy 2 2 1

.zm 2 2 1

.mk 2 2

.vc 2 2 1 1

.ls 1 1

.ga 1 1

.la 1 1 1 1

.gi 1 1 1

.me 1 1 1

.vg 1 1 1

.cg 1 1

.as 1 1

.sc 1 1 1

.bj 1 1

.gm 1 1 1

.ad 1 1 1

.sm 1 1 1

.dj 1 1

.gg 1 1

.ki 1 1

.biz 1 1

.gl 1 1

.nu 1 1 1

.st 1 1

.ck 1 1

.ao 1 1

.tm 1 1

.pn 1 1

.sb 1 1

.ai 1 1

.ir 1 1

.sl 1 1

.tv 1 1

.bi 1 1

.gd 1 1

.tk 1 1

.nr 1 1

.vu 1 1

.edu 1 1

.mw 1 1

.to 1 1

.yu 1 1 1 1

.eu 1 1

.ac 1 1

.je 1 1

.cc 1 1

.sh 1 1 1

.tl 1 1 1

.tj 1 1

.im 1 1

.fm 1 1 1

.ws 1 1 1

.tg 1 1

undefined 41,014 25,192 4,231 11,591

Total 138,457 119,148 7,091 12,200

In alphabetical order

Requests originating from a Google ip address

Service Total Pages Images Other

Wireless 3,257 2,220 1,022 15

Web search 9 8 1 1

Translate 467 466 1

Toolbar 1 1

Other 315 225 31 60

Maps 1 1 1

KeywordTool 1 1

Image search 1 1

GoogleBot 23,269 20,309 433 2,527

FeedFetcher 275 95 1 180

Desktop 1 1

Total 27,592 23,323 1,487 2,782

Requests originating from elsewhere

Service Total Pages Images Other

Wireless 12 12 1

Web search 93,313 91,663 1,121 529

Translate 26 6 11 9

Toolbar 965 169 53 744

Other 4,726 152 2,111 2,462

Maps 359 56 299 4

Mail 59 10 43 6

Image search 3,167 1,793 1,330 44

GoogleBot? 1,625 1,528 5 92

FeedFetcher 1 1 1

Earth 743 102 639 2

Desktop 5,873 339 1 5,534

Total 110,868 95,830 5,612 9,426

Top level domains

.ac 1 1

.ad 1 1 1

.ae 113 110 3 1

.af 2 2 1

.ag 2 2

.ai 1 1

.am 7 6 1 1

.ao 1 1

.ar 808 784 19 5

.as 1 1

.at 691 664 24 3

.au 1,326 1,280 36 9

.az 12 11 1 1

.ba 78 74 3 1

.bd 34 32 1 1

.be 709 688 19 2

.bg 287 279 7 2

.bh 23 23 1 1

.bi 1 1

.biz 1 1

.bj 1 1

.bn 7 6 1 1

.bo 51 49 1 1

.br 2,665 2,601 45 19

.bs 7 7 1

.bw 3 3 1 1

.by 38 37 1 1

.bz 3 3 1 1

.ca 3,148 3,049 85 14

.cat 7 7 1 1

.cc 1 1

.cd 2 2 1 1

.cg 1 1

.ch 648 629 16 2

.ci 7 7 1 1

.ck 1 1

.cl 532 518 11 3

.cn 357 312 42 3

.co 501 481 9 12

.com 34,206 32,703 1,222 282

.cr 70 69 1 1

.cu 12 11 1 1

.cz 261 248 12 2

.de 7,405 7,174 204 27

.dj 1 1

.dk 333 321 11 1

.dm 3 3 1

.do 136 133 2 1

.dz 8 8 1 1

.ec 199 194 3 1

.edu 1 1

.ee 84 81 2 1

.eg 186 179 6 2

.es 2,448 2,381 57 10

.et 12 11 1 1

.eu 1 1

.fi 666 653 11 2

.fj 3 3 1 1

.fm 1 1 1

.fr 4,578 4,389 151 38

.ga 1 1

.gd 1 1

.ge 20 18 2 1

.gg 1 1

.gh 12 12 1 1

.gi 1 1 1

.gl 1 1

.gm 1 1 1

.gp 4 4 1 1

.gr 308 298 8 1

.gt 48 46 2 1

.gy 2 2 1

.hk 183 178 4 1

.hn 29 28 1 1

.hr 231 223 6 1

.ht 2 2 1 1

.hu 473 448 20 6

.id 536 510 20 6

.ie 284 277 6 1

.il 420 411 7 2

.im 1 1

.in 1,762 1,699 51 11

.ir 1 1

.is 25 24 1 1

.it 3,349 3,268 69 11

.je 1 1

.jm 28 28 1 1

.jo 50 48 2 1

.jp 4,512 4,420 71 21

.ke 29 29 1 1

.kg 4 4 1 1

.kh 4 3 1

.ki 1 1

.kr 97 79 18 1

.kw 32 31 1 1

.kz 32 31 1 1

.la 1 1 1 1

.lb 20 19 1 1

.li 2 2 1

.lk 43 41 1 1

.ls 1 1

.lt 181 175 4 1

.lu 27 26 1 1

.lv 91 88 3 1

.ly 12 12 1 1

.ma 145 141 2 1

.md 35 34 1 1

.me 1 1 1

.mg 3 3 1

.mk 2 2

.mn 10 9 1 1

.mt 25 24 1 1

.mu 15 14 1 1

.mv 6 5 1 1

.mw 1 1

.mx 2,259 2,208 43 8

.my 349 338 9 2

.mz 3 3

.na 2 1 1

.net 3 1 3 1

.ng 29 28 1 1

.ni 16 15 1 1

.nl 1,208 1,167 38 3

.no 325 312 11 1

.np 12 11 1 1

.nr 1 1

.nu 1 1 1

.nz 235 217 5 12

.om 20 20 1 1

.org 3 1 2 1

.pa 7 7 1 1

.pe 566 555 9 3

.ph 599 585 9 6

.pk 174 168 5 1

.pl 3,426 3,356 57 13

.pn 1 1

.pr 90 88 2 1

.ps 18 17 1 1

.pt 491 478 11 2

.py 31 30 1 1

.qa 26 26 1 1

.ro 432 412 17 2

.rs 36 35 1 1

.ru 1,093 1,020 68 5

.rw 2 2 1 1

.sa 119 114 4 1

.sb 1 1

.sc 1 1 1

.se 845 822 21 2

.sg 309 303 5 1

.sh 1 1 1

.si 142 137 4 1

.sk 171 161 7 3

.sl 1 1

.sm 1 1 1

.sn 9 9 1 1

.st 1 1

.sv 46 45 1 1

.tg 1 1

.th 560 543 13 3

.tj 1 1

.tk 1 1

.tl 1 1 1

.tm 1 1

.to 1 1

.tr 1,363 1,316 41 6

.tt 27 27 1 1

.tv 1 1

.tw 229 220 7 2

.tz 7 6 1

.ua 399 391 6 2

.ug 2 2

.uk 4,925 4,777 130 18

.uy 61 59 2 1

.uz 3 3 1 1

.vc 2 2 1 1

.ve 482 473 6 3

.vg 1 1 1

.vi 2 2 1 1

.vn 370 361 7 2

.vu 1 1

.ws 1 1 1

.yu 1 1 1 1

.za 125 122 3 1

.zm 2 2 1

.zw 3 3 1 1

undefined 41,014 25,192 4,231 11,591

Total 138,457 119,148 7,091 12,200

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges
Agents: as for genuine agent strings: too many crawlers indentify themselves as 'GoogleBot' to take this at face value. They are accepted as genuine Google crawler requests only when the ip address matches a known range (see above). Other records that mention GoogleBot are counted as GoogleBot? (question mark, as this may include partners, like DoCoMo). However when the agent string mentions Google Desktop or Google Earth this is always accepted
Service: the service name is based on the agent string (plus for GoogleBot check for ip address, see above), if this is inconclusive it is based on the referer string.
Here is detailed breakdown per service of indicators that pointed to Google (total ≥ 3)

Service Total Originating from
Google ip address Referer mentions
Google url Agent mentions
Google service

Desktop 5,873 - - Y
Earth 743 - - Y
FeedFetcher 275 Y - Y
GoogleBot 23,268 Y - Y
GoogleBot? 1,623 - - Y
Image search 3,167 - Y -
Mail 59 - Y -
Maps 359 - Y -
Other 44 - - Y
Other 4,667 - Y -
Other 15 - Y Y
Other 60 Y - -
Other 255 Y - Y
Toolbar 823 - - Y
Toolbar 100 - Y -
Toolbar 42 - Y Y
Translate 25 - Y -
Translate 326 Y - Y
Translate 141 Y Y Y
Web search 93,313 - Y -
Web search 9 Y Y -
Wireless 12 - - Y
Wireless 3,257 Y - Y

Top Level Domain 'undefined': requests with top level domain 'undefined' are nearly all requests from anonymous ip addresses (crawler and other services)
Note: averages below 1 are always rounded up to 1

Generated on Fri, Feb 26, 2010 13:49
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.