Wikimedia Traffic Analysis Report

Wikimedia Traffic Analysis Report - Google requests

Daily averages, based on sample period: 1 Sep 2010 - 30 Sep 2010

This analysis is based on a 1:1000 sampled server log (squids) ⇒ all counts x 1000.
See also: Requests by destination or by origin / Methods / Scripts / Skins / Crawlers / Op.Sys. / Browsers / Google

This report shows all requests to Wikimedia servers where a Google server of service was involved in any way, be it the GoogleBot crawler or FeedFetcher collector scripts that run on Google servers, or a user that follows a link from a Google Web or Google Desktop search results page, or from Google Maps or Google Earth etcetera.
Technically speaking three fields in the squid log records are checked for this: client ip address, referer header and user agent header. A request can originate from an ip address which has been registered by Google and/or it can carry a referer tag that tells us a user clicked a link on a Google results page and/or it can carry an agent string that mentions a Google application which can reasonably be assumed to be genuinely Google's. See bottom of page for further details.
In total Google was somehow involved in 43.6% of daily external page^* requests
Google referred to our sites, through its services including search, maps, and Google Earth, 147,418,000 page views per day, representing 39.6% of our external page requests.
Including all of its different search crawlers and services hosted on its servers, Google itself requested another 14,955,000 page pages per day, representing 4.0% of our external page requests.
* = mime type text/html only

In order of request volume

Requests originating from a Google ip address

Service Total Pages Images Other

GoogleBot 11,076 9,838 538 700

Wireless 3,921 3,863 55 3

Translate 888 888

Other 527 262 34 231

FeedFetcher 308 91 11 205

Web search 14 13 1 1

Toolbar 1 1 1

Total 16,734 14,955 638 1,140

Requests originating from elsewhere

Service Total Pages Images Other

Web search 149,666 143,978 2,659 3,028

GoogleBot? 3,744 3,015 29 700

Desktop 1,017 43 1 974

Toolbar 311 125 45 142

Other 248 66 53 129

Earth 221 89 132 1

Image search 111 56 55 1

Maps 108 34 72 1

Translate 26 4 16 6

Mail 13 5 7 1

Wireless 6 3 3

FeedFetcher 1 1 1

Total 155,471 147,418 3,071 4,981

Top level domains

.com 43,095 41,331 1,098 666

.de 11,377 11,118 162 96

.uk 8,413 8,194 140 79

.jp 7,672 6,927 67 678

.fr 6,786 6,611 77 98

.mx 6,267 6,092 65 109

.br 6,044 5,890 66 88

.it 5,674 5,525 73 75

.ca 4,796 4,660 73 63

.pl 4,608 4,510 60 37

.es 3,934 3,793 61 81

.in 3,235 3,006 65 164

.au 2,829 2,766 43 20

.co 2,324 2,261 21 42

.nl 1,935 1,875 41 20

.ar 1,732 1,670 25 38

.ru 1,650 1,556 73 20

.ph 1,321 1,278 24 19

.se 1,227 1,199 19 9

.pe 1,220 1,136 14 70

.tr 1,176 1,112 26 38

.cl 1,157 1,118 17 23

.th 1,097 924 123 50

.ch 1,056 1,023 22 11

.be 1,056 1,022 20 14

.at 938 911 15 12

.fi 932 916 9 7

.id 826 770 45 11

.hu 722 692 20 10

.ua 720 702 8 10

.pt 679 659 11 8

.my 637 598 31 7

.vn 612 525 44 43

.ro 603 570 15 17

.ve 594 576 7 11

.hk 583 513 27 42

.dk 577 558 13 5

.il 557 532 7 17

.no 525 512 10 3

.sg 509 491 14 4

.gr 502 486 10 6

.nz 484 461 8 15

.ec 476 461 4 12

.ie 447 437 5 5

.do 420 407 3 9

.bg 385 368 7 10

.cz 384 367 13 4

.hr 342 329 8 5

.za 310 297 7 6

.tw 291 272 5 13

.lt 265 257 5 3

.gt 259 249 2 7

.pk 254 235 6 13

.sk 236 222 9 5

.eg 234 212 6 16

.ma 191 176 3 12

.si 181 173 5 3

.cr 180 173 3 3

.pr 175 171 3 1

.sv 172 166 1 4

.uy 171 165 2 3

.bo 170 160 2 8

.sa 170 160 3 8

.ae 160 156 3 1

.kr 156 115 21 20

.lv 125 120 3 1

.ee 124 121 2 1

.ba 104 100 2 2

.hn 89 85 1 3

.py 88 86 1 1

.lk 77 73 3 2

.by 66 65 1 1

.ke 63 61 1 1

.ng 62 59 1 3

.kz 61 60 1 1

.rs 57 54 1 2

.md 56 55 1 1

.jo 55 51 1 2

.jm 53 52 1 1

.tt 51 50 1 1

.ni 46 45 1 1

.kw 42 41 1 1

.is 39 38 1 1

.qa 39 37 1 1

.lu 37 36 1 1

.ge 33 31 2 1

.mt 31 31 1 1

.gh 31 30 1 1

.lb 30 29 1 1

.mu 29 28 1 1

.bh 28 28 1 1

.ps 24 22 1 2

.pa 23 23 1 1

.np 23 21 1 1

.az 21 19 1 1

.ug 19 18 1 1

.cu 19 18 1 1

.org 19 2 17 1

.om 18 17 1 1

.am 18 18 1 1

.ly 16 14 1 1

.cm 16 16 1 1

.ci 14 13 1 1

.bs 13 13 1 1

.mn 12 11 1 1

.mz 12 11 1 1

.tz 11 10 1 1

.cat 11 11 1 1

.bn 11 11 1 1

.mv 10 9 1 1

.fj 10 10 1 1

.sn 10 9 1 1

.zw 10 9 1 1

.et 9 8 1 1

.bw 8 8 1 1

.bz 8 8 1

.net 8 2 6 1

.na 7 7 1 1

.mg 7 7 1 1

.rw 6 5 1

.gy 6 6 1 1

.bd 5 4 1 1

.dz 5 4 1 1

.kh 5 5 1 1

.zm 5 5 1 1

.dm 4 4 1 1

.kg 4 4 1

.cd 4 3 1

.ag 4 4 1

.vi 4 4 1

.uz 4 4 1 1

.vc 4 4

.gp 4 4 1 1

.li 3 3 1

.cn 3 3 1

.ht 3 3 1 1

.mk 3 3 1 1

.bf 2 2 1 1

.af 2 2 1 1

.ls 2 2

.la 2 2 1

.gi 2 2

.ai 2 1 1 1

.me 2 2 1 1

.vg 2 2 1 1

.as 2 1 1 1

.ml 2 2 1 1

.ne 1 1 1 1

.biz 1 1 1 1

.gl 1 1

.ga 1 1 1

.cg 1 1 1

.us 1 1 1

.bi 1 1 1 1

.sc 1 1

.bj 1 1 1 1

.gm 1 1 1 1

.to 1 1 1

.tl 1 1 1 1

.ws 1 1 1 1

.ad 1 1 1 1

.tg 1 1 1

.sm 1 1 1 1

.dj 1 1 1 1

.gg 1 1 1

.ki 1 1

.nu 1 1

.st 1 1

.ck 1 1

.ao 1 1

.tm 1 1

.ms 1 1

.pn 1 1

.sb 1 1 1

.gs 1 1

.ir 1 1

.sl 1 1 1 1

.tv 1 1 1

.tk 1 1

.su 1 1

.nr 1 1

.vu 1 1

.edu 1 1

.mw 1 1

.eu 1 1 1

.je 1 1 1

.cc 1 1

.tj 1 1

.im 1 1

.fm 1 1

undefined 21,475 17,657 752 3,067

Total 172,201 162,368 3,703 6,115

In alphabetical order

Requests originating from a Google ip address

Service Total Pages Images Other

Wireless 3,921 3,863 55 3

Web search 14 13 1 1

Translate 888 888

Toolbar 1 1 1

Other 527 262 34 231

GoogleBot 11,076 9,838 538 700

FeedFetcher 308 91 11 205

Total 16,734 14,955 638 1,140

Requests originating from elsewhere

Service Total Pages Images Other

Wireless 6 3 3

Web search 149,666 143,978 2,659 3,028

Translate 26 4 16 6

Toolbar 311 125 45 142

Other 248 66 53 129

Maps 108 34 72 1

Mail 13 5 7 1

Image search 111 56 55 1

GoogleBot? 3,744 3,015 29 700

FeedFetcher 1 1 1

Earth 221 89 132 1

Desktop 1,017 43 1 974

Total 155,471 147,418 3,071 4,981

Top level domains

.ad 1 1 1 1

.ae 160 156 3 1

.af 2 2 1 1

.ag 4 4 1

.ai 2 1 1 1

.am 18 18 1 1

.ao 1 1

.ar 1,732 1,670 25 38

.as 2 1 1 1

.at 938 911 15 12

.au 2,829 2,766 43 20

.az 21 19 1 1

.ba 104 100 2 2

.bd 5 4 1 1

.be 1,056 1,022 20 14

.bf 2 2 1 1

.bg 385 368 7 10

.bh 28 28 1 1

.bi 1 1 1 1

.biz 1 1 1 1

.bj 1 1 1 1

.bn 11 11 1 1

.bo 170 160 2 8

.br 6,044 5,890 66 88

.bs 13 13 1 1

.bw 8 8 1 1

.by 66 65 1 1

.bz 8 8 1

.ca 4,796 4,660 73 63

.cat 11 11 1 1

.cc 1 1

.cd 4 3 1

.cg 1 1 1

.ch 1,056 1,023 22 11

.ci 14 13 1 1

.ck 1 1

.cl 1,157 1,118 17 23

.cm 16 16 1 1

.cn 3 3 1

.co 2,324 2,261 21 42

.com 43,095 41,331 1,098 666

.cr 180 173 3 3

.cu 19 18 1 1

.cz 384 367 13 4

.de 11,377 11,118 162 96

.dj 1 1 1 1

.dk 577 558 13 5

.dm 4 4 1 1

.do 420 407 3 9

.dz 5 4 1 1

.ec 476 461 4 12

.edu 1 1

.ee 124 121 2 1

.eg 234 212 6 16

.es 3,934 3,793 61 81

.et 9 8 1 1

.eu 1 1 1

.fi 932 916 9 7

.fj 10 10 1 1

.fm 1 1

.fr 6,786 6,611 77 98

.ga 1 1 1

.ge 33 31 2 1

.gg 1 1 1

.gh 31 30 1 1

.gi 2 2

.gl 1 1

.gm 1 1 1 1

.gp 4 4 1 1

.gr 502 486 10 6

.gs 1 1

.gt 259 249 2 7

.gy 6 6 1 1

.hk 583 513 27 42

.hn 89 85 1 3

.hr 342 329 8 5

.ht 3 3 1 1

.hu 722 692 20 10

.id 826 770 45 11

.ie 447 437 5 5

.il 557 532 7 17

.im 1 1

.in 3,235 3,006 65 164

.ir 1 1

.is 39 38 1 1

.it 5,674 5,525 73 75

.je 1 1 1

.jm 53 52 1 1

.jo 55 51 1 2

.jp 7,672 6,927 67 678

.ke 63 61 1 1

.kg 4 4 1

.kh 5 5 1 1

.ki 1 1

.kr 156 115 21 20

.kw 42 41 1 1

.kz 61 60 1 1

.la 2 2 1

.lb 30 29 1 1

.li 3 3 1

.lk 77 73 3 2

.ls 2 2

.lt 265 257 5 3

.lu 37 36 1 1

.lv 125 120 3 1

.ly 16 14 1 1

.ma 191 176 3 12

.md 56 55 1 1

.me 2 2 1 1

.mg 7 7 1 1

.mk 3 3 1 1

.ml 2 2 1 1

.mn 12 11 1 1

.ms 1 1

.mt 31 31 1 1

.mu 29 28 1 1

.mv 10 9 1 1

.mw 1 1

.mx 6,267 6,092 65 109

.my 637 598 31 7

.mz 12 11 1 1

.na 7 7 1 1

.ne 1 1 1 1

.net 8 2 6 1

.ng 62 59 1 3

.ni 46 45 1 1

.nl 1,935 1,875 41 20

.no 525 512 10 3

.np 23 21 1 1

.nr 1 1

.nu 1 1

.nz 484 461 8 15

.om 18 17 1 1

.org 19 2 17 1

.pa 23 23 1 1

.pe 1,220 1,136 14 70

.ph 1,321 1,278 24 19

.pk 254 235 6 13

.pl 4,608 4,510 60 37

.pn 1 1

.pr 175 171 3 1

.ps 24 22 1 2

.pt 679 659 11 8

.py 88 86 1 1

.qa 39 37 1 1

.ro 603 570 15 17

.rs 57 54 1 2

.ru 1,650 1,556 73 20

.rw 6 5 1

.sa 170 160 3 8

.sb 1 1 1

.sc 1 1

.se 1,227 1,199 19 9

.sg 509 491 14 4

.si 181 173 5 3

.sk 236 222 9 5

.sl 1 1 1 1

.sm 1 1 1 1

.sn 10 9 1 1

.st 1 1

.su 1 1

.sv 172 166 1 4

.tg 1 1 1

.th 1,097 924 123 50

.tj 1 1

.tk 1 1

.tl 1 1 1 1

.tm 1 1

.to 1 1 1

.tr 1,176 1,112 26 38

.tt 51 50 1 1

.tv 1 1 1

.tw 291 272 5 13

.tz 11 10 1 1

.ua 720 702 8 10

.ug 19 18 1 1

.uk 8,413 8,194 140 79

.us 1 1 1

.uy 171 165 2 3

.uz 4 4 1 1

.vc 4 4

.ve 594 576 7 11

.vg 2 2 1 1

.vi 4 4 1

.vn 612 525 44 43

.vu 1 1

.ws 1 1 1 1

.za 310 297 7 6

.zm 5 5 1 1

.zw 10 9 1 1

undefined 21,475 17,657 752 3,067

Total 172,201 162,368 3,703 6,115

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges
Agents: as for genuine agent strings: too many crawlers indentify themselves as 'GoogleBot' to take this at face value. They are accepted as genuine Google crawler requests only when the ip address matches a known range (see above). Other records that mention GoogleBot are counted as GoogleBot? (question mark, as this may include partners, like DoCoMo). However when the agent string mentions Google Desktop or Google Earth this is always accepted
Service: the service name is based on the agent string (plus for GoogleBot check for ip address, see above), if this is inconclusive it is based on the referer string.
Here is detailed breakdown per service of indicators that pointed to Google (total ≥ 3)

Service Total Originating from
Google ip address Referer mentions
Google url Agent mentions
Google service

Desktop 1,017 - - Y
Earth 221 - - Y
FeedFetcher 308 Y - Y
GoogleBot 11,076 Y - Y
GoogleBot? 3,736 - - Y
GoogleBot? 9 - Y Y
Image search 111 - Y -
Mail 13 - Y -
Maps 108 - Y -
Other 194 - - Y
Other 7 - Y -
Other 46 - Y Y
Other 274 Y - -
Other 252 Y - Y
Toolbar 209 - - Y
Toolbar 67 - Y -
Toolbar 35 - Y Y
Translate 25 - Y -
Translate 672 Y - Y
Translate 216 Y Y Y
Web search 149,666 - Y -
Web search 14 Y Y -
Wireless 6 - - Y
Wireless 3,921 Y - Y

Top Level Domain 'undefined': requests with top level domain 'undefined' are nearly all requests from anonymous ip addresses (crawler and other services)
Note: averages below 1 are always rounded up to 1

Generated on Tue, Oct 19, 2010 3:21
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.