Wikimedia Visitor Log Analysis Report

Wikimedia Visitor Log Analysis Report - Google requests

Daily averages, based on sample period: 1 Jan 2010 - 31 Jan 2010

This analysis is based on a 1:1000 sampled server log (squids) ⇒ all counts x 1000.
See also: Requests by destination or by origin / Methods / Scripts / Skins / Crawlers / Op.Sys. / Browsers / Google

This report shows all requests to Wikimedia servers where a Google server of service was involved in any way, be it the GoogleBot crawler or FeedFetcher collector scripts that run on Google servers, or a user that follows a link from a Google Web or Google Desktop search results page, or from Google Maps or Google Earth etcetera.
Technically speaking three fields in the squid log records are checked for this: client ip address, referer header and user agent header. A request can originate from an ip address which has been registered by Google and/or it can carry a referer tag that tells us a user clicked a link on a Google results page and/or it can carry an agent string that mentions a Google application which can reasonably be assumed to be genuinely Google's. See bottom of page for further details.
In total Google was somehow involved in 51.6% of daily external page^* requests
Google referred to our sites, through its services including search, maps, and Google Earth, 106,290,000 page views per day, representing 41.9% of our external page requests.
Including all of its different search crawlers and services hosted on its servers, Google itself requested another 24,493,000 page pages per day, representing 9.7% of our external page requests.
* = mime type text/html only

In order of request volume

Requests originating from a Google ip address

Service Total Pages Images Other

GoogleBot 23,751 21,103 381 2,267

Wireless 3,012 2,554 449 8

Translate 542 542 1

Other 268 194 31 43

FeedFetcher 259 91 1 167

Web search 9 9 1 1

Desktop 1 1

Image search 1 1

Toolbar 1 1

Maps 1 1

KeywordTool 1 1

Total 27,842 24,493 862 2,486

Requests originating from elsewhere

Service Total Pages Images Other

Web search 104,966 102,095 1,081 1,790

Desktop 5,389 248 1 5,141

Other 4,341 142 1,975 2,224

Image search 3,444 1,990 1,404 51

GoogleBot? 1,542 1,455 5 82

Toolbar 953 169 45 739

Earth 798 100 695 3

Maps 405 60 341 4

Mail 33 6 23 4

Translate 23 4 11 8

Wireless 21 21 1

FeedFetcher 1 1 1

Total 121,916 106,290 5,580 10,046

Top level domains

.com 37,732 35,023 1,217 1,493

.de 8,801 8,537 222 42

.uk 5,901 5,719 154 28

.fr 5,224 5,050 137 37

.jp 4,929 4,831 78 20

.pl 4,090 4,009 62 20

.it 4,074 3,976 78 19

.ca 3,561 3,455 93 13

.mx 2,966 2,910 47 8

.es 2,855 2,779 63 13

.br 2,439 2,385 44 10

.in 1,800 1,739 49 11

.au 1,447 1,405 36 6

.nl 1,397 1,351 41 5

.ru 1,069 1,000 64 5

.tr 1,051 1,014 31 6

.se 957 927 26 4

.ph 909 885 14 9

.be 838 814 21 3

.at 806 776 25 6

.ve 764 752 8 4

.ch 760 738 19 3

.fi 743 728 12 3

.co 737 713 10 13

.ar 662 644 15 3

.th 620 596 20 4

.pt 619 603 13 3

.id 616 579 31 6

.ro 596 574 18 4

.hu 584 558 20 6

.cl 439 428 9 3

.pe 424 417 5 1

.my 409 395 11 3

.il 405 396 7 3

.no 403 389 12 2

.ua 398 390 5 3

.dk 391 376 13 2

.vn 369 356 10 2

.gr 357 346 9 2

.sg 357 350 6 2

.bg 343 334 8 2

.cn 334 304 26 3

.ie 330 323 6 2

.cz 326 312 12 2

.hr 263 253 8 2

.nz 256 244 6 7

.ec 248 244 3 2

.sk 219 206 9 4

.tw 219 212 5 2

.lt 202 196 4 2

.hk 188 184 3 1

.za 173 168 5 1

.pk 173 168 4 1

.do 171 168 2 1

.si 158 152 5 1

.eg 135 129 6 1

.ma 133 130 2 1

.gt 126 124 2 1

.sa 117 112 3 1

.ae 106 103 3 1

.lv 101 97 3 1

.pr 100 98 2 1

.ee 96 93 2 1

.sv 93 91 1 1

.kr 89 75 14 1

.cr 85 83 1 1

.ba 76 73 2 1

.uy 52 51 1 1

.lk 47 45 2 1

.bo 44 43 1 1

.rs 40 38 1 1

.by 37 36 1 1

.hn 35 34 1 1

.bd 34 33 1 1

.ke 34 33 1 1

.jo 34 33 1 1

.tt 34 34 1 1

.jm 33 32 1 1

.md 33 32 1 1

.ng 32 32 1 1

.py 31 31 1 1

.lu 30 29 1 1

.kz 30 29 1 1

.kw 30 29 1 1

.is 29 28 1 1

.qa 28 28 1 1

.mt 26 25 1 1

.lb 25 24 1 1

.bh 22 21 1

.ni 20 19 1 1

.ge 19 18 1 1

.om 18 18 1

.mu 16 16 1 1

.ps 13 12 1

.gh 13 13 1 1

.ly 12 11 1 1

.az 12 11 1 1

.cu 12 12 1 1

.sn 11 10 1 1

.np 11 10 1 1

.ci 9 9 1 1

.mn 9 8 1 1

.bs 9 8 1 1

.tz 8 7 1 1

.am 8 7 1

.dz 8 8 1 1

.cat 8 8 1 1

.et 8 7 1

.bn 8 8 1 1

.mv 7 7 1

.pa 6 6 1

.bw 5 4 1 1

.bz 5 5 1 1

.net 5 1 4 1

.gp 5 5 1 1

.fj 4 4 1 1

.mg 4 4 1

.kh 4 4 1

.org 4 1 3 1

.zw 4 4 1

.dm 3 3

.kg 3 3 1

.ag 3 3 1 1

.vi 3 3 1

.mz 3 3 1 1

.uz 3 3 1

.gy 3 3 1

.zm 3 3 1 1

.vc 3 3 1

.ug 2 2

.rw 2 2

.af 2 2 1 1

.ga 2 2

.gi 2 2

.na 2 2 1

.li 2 2 1

.cd 2 2 1

.ht 2 2

.gl 1 1 1

.ls 1 1

.la 1 1

.ai 1 1

.vg 1 1 1

.cg 1 1

.sl 1 1 1 1

.as 1 1 1

.sc 1 1 1

.bj 1 1

.gm 1 1 1

.mk 1 1 1

.ad 1 1

.sm 1 1 1

.dj 1 1

.gg 1 1

.ki 1 1

.biz 1 1

.nu 1 1

.st 1 1

.ck 1 1

.tm 1 1

.ms 1 1

.pn 1 1

.sb 1 1

.me 1 1 1

.gs 1 1

.ir 1 1 1

.td 1 1

.tv 1 1 1

.us 1 1 1

.bi 1 1

.gd 1 1

.tk 1 1 1

.su 1 1

.nr 1 1

.ml 1 1

.vu 1 1

.gf 1 1

.edu 1 1 1

.nf 1 1

.mw 1 1

.to 1 1

.eu 1 1 1

.ac 1 1

.je 1 1

.cc 1 1

.sh 1 1

.tl 1 1 1

.tj 1 1

.im 1 1 1

.fm 1 1

.ws 1 1 1

.tg 1 1

undefined 40,311 26,146 3,510 10,655

Total 149,755 130,779 6,439 12,525

In alphabetical order

Requests originating from a Google ip address

Service Total Pages Images Other

Wireless 3,012 2,554 449 8

Web search 9 9 1 1

Translate 542 542 1

Toolbar 1 1

Other 268 194 31 43

Maps 1 1

KeywordTool 1 1

Image search 1 1

GoogleBot 23,751 21,103 381 2,267

FeedFetcher 259 91 1 167

Desktop 1 1

Total 27,842 24,493 862 2,486

Requests originating from elsewhere

Service Total Pages Images Other

Wireless 21 21 1

Web search 104,966 102,095 1,081 1,790

Translate 23 4 11 8

Toolbar 953 169 45 739

Other 4,341 142 1,975 2,224

Maps 405 60 341 4

Mail 33 6 23 4

Image search 3,444 1,990 1,404 51

GoogleBot? 1,542 1,455 5 82

FeedFetcher 1 1 1

Earth 798 100 695 3

Desktop 5,389 248 1 5,141

Total 121,916 106,290 5,580 10,046

Top level domains

.ac 1 1

.ad 1 1

.ae 106 103 3 1

.af 2 2 1 1

.ag 3 3 1 1

.ai 1 1

.am 8 7 1

.ar 662 644 15 3

.as 1 1 1

.at 806 776 25 6

.au 1,447 1,405 36 6

.az 12 11 1 1

.ba 76 73 2 1

.bd 34 33 1 1

.be 838 814 21 3

.bg 343 334 8 2

.bh 22 21 1

.bi 1 1

.biz 1 1

.bj 1 1

.bn 8 8 1 1

.bo 44 43 1 1

.br 2,439 2,385 44 10

.bs 9 8 1 1

.bw 5 4 1 1

.by 37 36 1 1

.bz 5 5 1 1

.ca 3,561 3,455 93 13

.cat 8 8 1 1

.cc 1 1

.cd 2 2 1

.cg 1 1

.ch 760 738 19 3

.ci 9 9 1 1

.ck 1 1

.cl 439 428 9 3

.cn 334 304 26 3

.co 737 713 10 13

.com 37,732 35,023 1,217 1,493

.cr 85 83 1 1

.cu 12 12 1 1

.cz 326 312 12 2

.de 8,801 8,537 222 42

.dj 1 1

.dk 391 376 13 2

.dm 3 3

.do 171 168 2 1

.dz 8 8 1 1

.ec 248 244 3 2

.edu 1 1 1

.ee 96 93 2 1

.eg 135 129 6 1

.es 2,855 2,779 63 13

.et 8 7 1

.eu 1 1 1

.fi 743 728 12 3

.fj 4 4 1 1

.fm 1 1

.fr 5,224 5,050 137 37

.ga 2 2

.gd 1 1

.ge 19 18 1 1

.gf 1 1

.gg 1 1

.gh 13 13 1 1

.gi 2 2

.gl 1 1 1

.gm 1 1 1

.gp 5 5 1 1

.gr 357 346 9 2

.gs 1 1

.gt 126 124 2 1

.gy 3 3 1

.hk 188 184 3 1

.hn 35 34 1 1

.hr 263 253 8 2

.ht 2 2

.hu 584 558 20 6

.id 616 579 31 6

.ie 330 323 6 2

.il 405 396 7 3

.im 1 1 1

.in 1,800 1,739 49 11

.ir 1 1 1

.is 29 28 1 1

.it 4,074 3,976 78 19

.je 1 1

.jm 33 32 1 1

.jo 34 33 1 1

.jp 4,929 4,831 78 20

.ke 34 33 1 1

.kg 3 3 1

.kh 4 4 1

.ki 1 1

.kr 89 75 14 1

.kw 30 29 1 1

.kz 30 29 1 1

.la 1 1

.lb 25 24 1 1

.li 2 2 1

.lk 47 45 2 1

.ls 1 1

.lt 202 196 4 2

.lu 30 29 1 1

.lv 101 97 3 1

.ly 12 11 1 1

.ma 133 130 2 1

.md 33 32 1 1

.me 1 1 1

.mg 4 4 1

.mk 1 1 1

.ml 1 1

.mn 9 8 1 1

.ms 1 1

.mt 26 25 1 1

.mu 16 16 1 1

.mv 7 7 1

.mw 1 1

.mx 2,966 2,910 47 8

.my 409 395 11 3

.mz 3 3 1 1

.na 2 2 1

.net 5 1 4 1

.nf 1 1

.ng 32 32 1 1

.ni 20 19 1 1

.nl 1,397 1,351 41 5

.no 403 389 12 2

.np 11 10 1 1

.nr 1 1

.nu 1 1

.nz 256 244 6 7

.om 18 18 1

.org 4 1 3 1

.pa 6 6 1

.pe 424 417 5 1

.ph 909 885 14 9

.pk 173 168 4 1

.pl 4,090 4,009 62 20

.pn 1 1

.pr 100 98 2 1

.ps 13 12 1

.pt 619 603 13 3

.py 31 31 1 1

.qa 28 28 1 1

.ro 596 574 18 4

.rs 40 38 1 1

.ru 1,069 1,000 64 5

.rw 2 2

.sa 117 112 3 1

.sb 1 1

.sc 1 1 1

.se 957 927 26 4

.sg 357 350 6 2

.sh 1 1

.si 158 152 5 1

.sk 219 206 9 4

.sl 1 1 1 1

.sm 1 1 1

.sn 11 10 1 1

.st 1 1

.su 1 1

.sv 93 91 1 1

.td 1 1

.tg 1 1

.th 620 596 20 4

.tj 1 1

.tk 1 1 1

.tl 1 1 1

.tm 1 1

.to 1 1

.tr 1,051 1,014 31 6

.tt 34 34 1 1

.tv 1 1 1

.tw 219 212 5 2

.tz 8 7 1 1

.ua 398 390 5 3

.ug 2 2

.uk 5,901 5,719 154 28

.us 1 1 1

.uy 52 51 1 1

.uz 3 3 1

.vc 3 3 1

.ve 764 752 8 4

.vg 1 1 1

.vi 3 3 1

.vn 369 356 10 2

.vu 1 1

.ws 1 1 1

.za 173 168 5 1

.zm 3 3 1 1

.zw 4 4 1

undefined 40,311 26,146 3,510 10,655

Total 149,755 130,779 6,439 12,525

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges
Agents: as for genuine agent strings: too many crawlers indentify themselves as 'GoogleBot' to take this at face value. They are accepted as genuine Google crawler requests only when the ip address matches a known range (see above). Other records that mention GoogleBot are counted as GoogleBot? (question mark, as this may include partners, like DoCoMo). However when the agent string mentions Google Desktop or Google Earth this is always accepted
Service: the service name is based on the agent string (plus for GoogleBot check for ip address, see above), if this is inconclusive it is based on the referer string.
Here is detailed breakdown per service of indicators that pointed to Google (total ≥ 3)

Service Total Originating from
Google ip address Referer mentions
Google url Agent mentions
Google service

Desktop 5,389 - - Y
Earth 798 - - Y
FeedFetcher 259 Y - Y
GoogleBot 23,751 Y - Y
GoogleBot? 1,541 - - Y
Image search 3,444 - Y -
Mail 33 - Y -
Maps 405 - Y -
Other 50 - - Y
Other 4,273 - Y -
Other 18 - Y Y
Other 63 Y - -
Other 205 Y - Y
Toolbar 818 - - Y
Toolbar 92 - Y -
Toolbar 43 - Y Y
Translate 23 - Y -
Translate 385 Y - Y
Translate 157 Y Y Y
Web search 104,966 - Y -
Web search 9 Y Y -
Wireless 21 - - Y
Wireless 3,012 Y - Y

Top Level Domain 'undefined': requests with top level domain 'undefined' are nearly all requests from anonymous ip addresses (crawler and other services)
Note: averages below 1 are always rounded up to 1

Generated on Thu, Mar 11, 2010 1:57
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.