Wikimedia Visitor Log Analysis Report

Wikimedia Visitor Log Analysis Report - Google requests

Daily averages, based on sample period: 1 Mar 2010 - 31 Mar 2010

This analysis is based on a 1:1000 sampled server log (squids) ⇒ all counts x 1000.
See also: Requests by destination or by origin / Methods / Scripts / Skins / Crawlers / Op.Sys. / Browsers / Google

This report shows all requests to Wikimedia servers where a Google server of service was involved in any way, be it the GoogleBot crawler or FeedFetcher collector scripts that run on Google servers, or a user that follows a link from a Google Web or Google Desktop search results page, or from Google Maps or Google Earth etcetera.
Technically speaking three fields in the squid log records are checked for this: client ip address, referer header and user agent header. A request can originate from an ip address which has been registered by Google and/or it can carry a referer tag that tells us a user clicked a link on a Google results page and/or it can carry an agent string that mentions a Google application which can reasonably be assumed to be genuinely Google's. See bottom of page for further details.
In total Google was somehow involved in 43.5% of daily external page^* requests
Google referred to our sites, through its services including search, maps, and Google Earth, 115,173,000 page views per day, representing 37.9% of our external page requests.
Including all of its different search crawlers and services hosted on its servers, Google itself requested another 17,243,000 page pages per day, representing 5.7% of our external page requests.
* = mime type text/html only

In order of request volume

Requests originating from a Google ip address

Service Total Pages Images Other

GoogleBot 14,841 13,134 792 915

Wireless 2,921 2,896 23 3

Translate 719 719 1 1

Other 508 402 43 63

FeedFetcher 271 84 10 177

Web search 9 8 1 1

Image search 1 1

Toolbar 1 1

Desktop 1 1

KeywordTool 1 1

Total 19,269 17,243 868 1,158

Requests originating from elsewhere

Service Total Pages Images Other

Web search 111,586 109,974 1,120 493

Image search 4,204 2,221 1,949 35

Other 3,869 163 2,029 1,678

Desktop 3,106 199 2,906

GoogleBot? 2,478 2,327 7 143

Toolbar 928 152 41 736

Earth 540 64 474 1

Maps 100 55 43 1

Translate 27 5 16 7

Wireless 11 11 1 1

Mail 10 2 6 1

FeedFetcher 1 1 1

Total 126,859 115,173 5,685 6,001

Top level domains

.com 35,214 34,059 953 202

.de 8,502 8,302 167 33

.uk 6,287 6,134 130 23

.fr 5,404 5,280 101 23

.jp 5,368 5,236 114 18

.br 4,294 4,227 56 12

.ca 4,180 4,084 87 10

.it 4,001 3,919 68 14

.mx 3,843 3,779 55 9

.pl 3,811 3,737 58 15

.es 3,080 3,001 67 12

.au 2,442 2,386 49 7

.in 2,000 1,946 46 8

.co 1,795 1,736 20 40

.nl 1,492 1,451 36 5

.ru 1,210 1,088 118 4

.ar 1,207 1,184 19 5

.tr 1,159 1,118 35 6

.se 980 954 21 5

.ve 886 872 10 3

.id 864 625 234 6

.pe 859 845 11 2

.be 854 837 15 3

.ph 849 808 34 6

.ch 809 792 14 2

.at 782 759 20 4

.fi 743 730 10 2

.cl 717 701 12 3

.my 574 489 82 3

.th 574 448 121 5

.pt 569 555 11 3

.hu 556 534 18 4

.vn 540 441 97 2

.ro 535 517 16 2

.ua 438 429 6 3

.dk 422 410 11 1

.nz 414 400 10 3

.sg 410 384 25 1

.hk 403 395 6 2

.il 399 389 7 2

.no 388 376 10 2

.gr 350 340 8 2

.ie 336 329 5 2

.bg 322 312 8 2

.cz 319 307 11 1

.cn 290 255 34 2

.hr 267 258 8 2

.tw 263 256 5 2

.ec 240 236 3 1

.za 233 227 5 1

.sk 218 206 9 3

.eg 210 204 5 1

.gt 207 203 3 1

.lt 205 200 4 1

.do 202 198 3 1

.pk 191 185 5 1

.si 151 146 4 1

.ma 140 137 2 1

.cr 138 135 2 1

.bo 136 134 2 1

.sa 136 130 5 1

.ae 135 132 3

.pr 135 133 2 1

.kr 135 100 34 1

.sv 133 131 2 1

.uy 118 116 2 1

.lv 99 96 2 1

.ee 96 94 2 1

.ba 83 79 3 1

.hn 71 69 1 1

.lk 56 54 2 1

.py 53 52 1 1

.jo 52 51 1 1

.ke 48 47 1 1

.ni 44 43 1 1

.rs 42 41 1 1

.ng 40 39 1 1

.by 39 38 1 1

.tt 39 39 1 1

.jm 38 38 1 1

.kz 38 38 1 1

.md 36 35 1 1

.kw 35 33 1 1

.is 33 32 1 1

.lu 31 31 1 1

.qa 31 30 1 1

.lb 27 26 1 1

.bh 26 25 1

.mt 25 24 1 1

.ge 24 23 1 1

.mu 20 20 1 1

.om 19 19 1 1

.gh 19 18 1 1

.ps 17 17 1

.pa 17 17 1 1

.az 14 13 1 1

.cu 14 14 1 1

.ug 13 13 1

.ly 13 13 1 1

.bd 12 12 1 1

.am 12 11 1 1

.mn 12 11 1 1

.np 12 12 1

.ci 11 10 1

.sn 11 11 1 1

.mz 10 10 1

.bs 10 9 1

.et 10 9 1 1

.mv 9 8 1

.cat 9 9 1

.bn 9 9 1 1

.fj 8 8 1 1

.tz 7 7 1 1

.bw 7 7 1 1

.zw 7 7 1 1

.net 6 1 5 1

.org 6 1 5 1

.na 5 4 1

.dz 5 5 1 1

.cm 5 5 1 1

.bz 5 4 1

.dm 4 4 1

.kg 4 4 1

.mg 4 4 1

.kh 4 4 1

.uz 4 3 1

.gp 4 4 1 1

.rw 3 3 1

.cd 3 3

.ag 3 3 1

.vi 3 3

.gy 3 3

.zm 3 2 1 1

.vc 3 3

.af 2 2 1

.li 2 2 1

.ml 2 2 1 1

.biz 1 1 1 1

.gl 1 1 1

.bf 1 1

.ls 1 1 1 1

.la 1 1 1

.gi 1 1 1

.ga 1 1 1

.ai 1 1 1

.me 1 1 1 1

.vg 1 1

.us 1 1 1 1

.bi 1 1

.as 1 1 1

.sc 1 1

.bj 1 1 1

.ht 1 1 1 1

.gm 1 1

.to 1 1 1

.mk 1 1 1

.ad 1 1 1

.tg 1 1 1

.sm 1 1

.dj 1 1 1

.gg 1 1

.ki 1 1 1

.nu 1 1 1

.st 1 1

.ck 1 1

.ao 1 1

.tm 1 1

.ms 1 1

.sb 1 1

.ir 1 1 1

.cg 1 1

.sl 1 1 1

.tv 1 1

.tk 1 1

.nr 1 1 1

.vu 1 1

.edu 1 1

.mw 1 1 1

.yu 1 1

.eu 1 1

.ac 1 1

.je 1 1

.cc 1 1 1

.sh 1 1 1

.tl 1 1

.tj 1 1

.im 1 1 1

.fm 1 1

.ws 1 1 1

undefined 29,545 19,584 3,350 6,610

Total 146,123 132,412 6,540 7,154

In alphabetical order

Requests originating from a Google ip address

Service Total Pages Images Other

Wireless 2,921 2,896 23 3

Web search 9 8 1 1

Translate 719 719 1 1

Toolbar 1 1

Other 508 402 43 63

KeywordTool 1 1

Image search 1 1

GoogleBot 14,841 13,134 792 915

FeedFetcher 271 84 10 177

Desktop 1 1

Total 19,269 17,243 868 1,158

Requests originating from elsewhere

Service Total Pages Images Other

Wireless 11 11 1 1

Web search 111,586 109,974 1,120 493

Translate 27 5 16 7

Toolbar 928 152 41 736

Other 3,869 163 2,029 1,678

Maps 100 55 43 1

Mail 10 2 6 1

Image search 4,204 2,221 1,949 35

GoogleBot? 2,478 2,327 7 143

FeedFetcher 1 1 1

Earth 540 64 474 1

Desktop 3,106 199 2,906

Total 126,859 115,173 5,685 6,001

Top level domains

.ac 1 1

.ad 1 1 1

.ae 135 132 3

.af 2 2 1

.ag 3 3 1

.ai 1 1 1

.am 12 11 1 1

.ao 1 1

.ar 1,207 1,184 19 5

.as 1 1 1

.at 782 759 20 4

.au 2,442 2,386 49 7

.az 14 13 1 1

.ba 83 79 3 1

.bd 12 12 1 1

.be 854 837 15 3

.bf 1 1

.bg 322 312 8 2

.bh 26 25 1

.bi 1 1

.biz 1 1 1 1

.bj 1 1 1

.bn 9 9 1 1

.bo 136 134 2 1

.br 4,294 4,227 56 12

.bs 10 9 1

.bw 7 7 1 1

.by 39 38 1 1

.bz 5 4 1

.ca 4,180 4,084 87 10

.cat 9 9 1

.cc 1 1 1

.cd 3 3

.cg 1 1

.ch 809 792 14 2

.ci 11 10 1

.ck 1 1

.cl 717 701 12 3

.cm 5 5 1 1

.cn 290 255 34 2

.co 1,795 1,736 20 40

.com 35,214 34,059 953 202

.cr 138 135 2 1

.cu 14 14 1 1

.cz 319 307 11 1

.de 8,502 8,302 167 33

.dj 1 1 1

.dk 422 410 11 1

.dm 4 4 1

.do 202 198 3 1

.dz 5 5 1 1

.ec 240 236 3 1

.edu 1 1

.ee 96 94 2 1

.eg 210 204 5 1

.es 3,080 3,001 67 12

.et 10 9 1 1

.eu 1 1

.fi 743 730 10 2

.fj 8 8 1 1

.fm 1 1

.fr 5,404 5,280 101 23

.ga 1 1 1

.ge 24 23 1 1

.gg 1 1

.gh 19 18 1 1

.gi 1 1 1

.gl 1 1 1

.gm 1 1

.gp 4 4 1 1

.gr 350 340 8 2

.gt 207 203 3 1

.gy 3 3

.hk 403 395 6 2

.hn 71 69 1 1

.hr 267 258 8 2

.ht 1 1 1 1

.hu 556 534 18 4

.id 864 625 234 6

.ie 336 329 5 2

.il 399 389 7 2

.im 1 1 1

.in 2,000 1,946 46 8

.ir 1 1 1

.is 33 32 1 1

.it 4,001 3,919 68 14

.je 1 1

.jm 38 38 1 1

.jo 52 51 1 1

.jp 5,368 5,236 114 18

.ke 48 47 1 1

.kg 4 4 1

.kh 4 4 1

.ki 1 1 1

.kr 135 100 34 1

.kw 35 33 1 1

.kz 38 38 1 1

.la 1 1 1

.lb 27 26 1 1

.li 2 2 1

.lk 56 54 2 1

.ls 1 1 1 1

.lt 205 200 4 1

.lu 31 31 1 1

.lv 99 96 2 1

.ly 13 13 1 1

.ma 140 137 2 1

.md 36 35 1 1

.me 1 1 1 1

.mg 4 4 1

.mk 1 1 1

.ml 2 2 1 1

.mn 12 11 1 1

.ms 1 1

.mt 25 24 1 1

.mu 20 20 1 1

.mv 9 8 1

.mw 1 1 1

.mx 3,843 3,779 55 9

.my 574 489 82 3

.mz 10 10 1

.na 5 4 1

.net 6 1 5 1

.ng 40 39 1 1

.ni 44 43 1 1

.nl 1,492 1,451 36 5

.no 388 376 10 2

.np 12 12 1

.nr 1 1 1

.nu 1 1 1

.nz 414 400 10 3

.om 19 19 1 1

.org 6 1 5 1

.pa 17 17 1 1

.pe 859 845 11 2

.ph 849 808 34 6

.pk 191 185 5 1

.pl 3,811 3,737 58 15

.pr 135 133 2 1

.ps 17 17 1

.pt 569 555 11 3

.py 53 52 1 1

.qa 31 30 1 1

.ro 535 517 16 2

.rs 42 41 1 1

.ru 1,210 1,088 118 4

.rw 3 3 1

.sa 136 130 5 1

.sb 1 1

.sc 1 1

.se 980 954 21 5

.sg 410 384 25 1

.sh 1 1 1

.si 151 146 4 1

.sk 218 206 9 3

.sl 1 1 1

.sm 1 1

.sn 11 11 1 1

.st 1 1

.sv 133 131 2 1

.tg 1 1 1

.th 574 448 121 5

.tj 1 1

.tk 1 1

.tl 1 1

.tm 1 1

.to 1 1 1

.tr 1,159 1,118 35 6

.tt 39 39 1 1

.tv 1 1

.tw 263 256 5 2

.tz 7 7 1 1

.ua 438 429 6 3

.ug 13 13 1

.uk 6,287 6,134 130 23

.us 1 1 1 1

.uy 118 116 2 1

.uz 4 3 1

.vc 3 3

.ve 886 872 10 3

.vg 1 1

.vi 3 3

.vn 540 441 97 2

.vu 1 1

.ws 1 1 1

.yu 1 1

.za 233 227 5 1

.zm 3 2 1 1

.zw 7 7 1 1

undefined 29,545 19,584 3,350 6,610

Total 146,123 132,412 6,540 7,154

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges
Agents: as for genuine agent strings: too many crawlers indentify themselves as 'GoogleBot' to take this at face value. They are accepted as genuine Google crawler requests only when the ip address matches a known range (see above). Other records that mention GoogleBot are counted as GoogleBot? (question mark, as this may include partners, like DoCoMo). However when the agent string mentions Google Desktop or Google Earth this is always accepted
Service: the service name is based on the agent string (plus for GoogleBot check for ip address, see above), if this is inconclusive it is based on the referer string.
Here is detailed breakdown per service of indicators that pointed to Google (total ≥ 3)

Service Total Originating from
Google ip address Referer mentions
Google url Agent mentions
Google service

Desktop 3,106 - - Y
Earth 540 - - Y
FeedFetcher 271 Y - Y
GoogleBot 14,841 Y - Y
GoogleBot? 2,475 - - Y
GoogleBot? 3 - Y Y
Image search 4,204 - Y -
Mail 10 - Y -
Maps 100 - Y -
Other 107 - - Y
Other 3,761 - Y -
Other 140 Y - -
Other 368 Y - Y
Toolbar 808 - - Y
Toolbar 83 - Y -
Toolbar 38 - Y Y
Translate 27 - Y -
Translate 515 Y - Y
Translate 204 Y Y Y
Web search 111,586 - Y -
Web search 9 Y Y -
Wireless 11 - - Y
Wireless 2,921 Y - Y

Top Level Domain 'undefined': requests with top level domain 'undefined' are nearly all requests from anonymous ip addresses (crawler and other services)
Note: averages below 1 are always rounded up to 1

Generated on Wed, May 19, 2010 11:02
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.