Wikimedia Traffic Analysis Report

Wikimedia Traffic Analysis Report - Google requests

Daily averages, based on sample period: 1 Jul 2010 - 31 Jul 2010

This analysis is based on a 1:1000 sampled server log (squids) ⇒ all counts x 1000.
See also: Requests by destination or by origin / Methods / Scripts / Skins / Crawlers / Op.Sys. / Browsers / Google

This report shows all requests to Wikimedia servers where a Google server of service was involved in any way, be it the GoogleBot crawler or FeedFetcher collector scripts that run on Google servers, or a user that follows a link from a Google Web or Google Desktop search results page, or from Google Maps or Google Earth etcetera.
Technically speaking three fields in the squid log records are checked for this: client ip address, referer header and user agent header. A request can originate from an ip address which has been registered by Google and/or it can carry a referer tag that tells us a user clicked a link on a Google results page and/or it can carry an agent string that mentions a Google application which can reasonably be assumed to be genuinely Google's. See bottom of page for further details.
In total Google was somehow involved in 38.6% of daily external page^* requests
Google referred to our sites, through its services including search, maps, and Google Earth, 90,490,000 page views per day, representing 34.4% of our external page requests.
Including all of its different search crawlers and services hosted on its servers, Google itself requested another 11,024,000 page pages per day, representing 4.2% of our external page requests.
* = mime type text/html only

In order of request volume

Requests originating from a Google ip address

Service Total Pages Images Other

GoogleBot 8,305 7,185 392 728

Wireless 3,029 2,997 31 2

Translate 557 557 1

Other 362 185 23 154

FeedFetcher 287 92 27 168

Web search 9 8 1 1

Toolbar 1 1

Desktop 1 1

KeywordTool 1 1

Total 12,549 11,024 473 1,053

Requests originating from elsewhere

Service Total Pages Images Other

Web search 92,116 87,892 2,032 2,192

GoogleBot? 2,438 2,281 13 144

Desktop 810 41 1 769

Toolbar 235 85 37 113

Other 198 73 25 99

Earth 154 26 127 1

Image search 64 26 38 1

Maps 61 27 33 1

Wireless 35 33 1 1

Translate 25 3 15 6

Mail 11 3 7 1

FeedFetcher 1 1 1

Total 96,147 90,490 2,328 3,326

Top level domains

.com 27,844 26,767 593 484

.de 7,170 6,981 112 77

.jp 6,016 5,414 91 511

.uk 5,230 5,098 73 60

.fr 3,628 3,511 48 69

.it 3,359 3,265 44 50

.br 3,267 3,171 38 58

.ca 3,031 2,942 41 47

.pl 2,338 2,283 30 25

.in 2,262 2,091 46 126

.es 2,101 2,020 33 49

.au 2,014 1,965 29 20

.mx 1,855 1,786 28 41

.co 1,361 1,320 12 29

.ar 1,158 1,113 16 30

.ph 1,132 1,067 46 18

.nl 1,049 1,018 16 15

.ru 902 805 85 13

.th 843 649 150 45

.pe 786 721 10 54

.id 784 570 205 8

.tr 757 714 17 26

.cl 748 723 11 15

.se 634 619 10 5

.ve 608 587 6 15

.ch 602 584 10 8

.be 598 581 9 9

.at 572 553 10 9

.my 553 460 85 8

.fi 525 517 5 4

.vn 447 344 73 30

.hk 418 359 27 33

.pt 416 405 6 5

.hu 403 386 11 6

.sg 399 368 27 4

.il 393 375 6 11

.ro 382 359 11 12

.nz 366 336 6 24

.ua 350 340 4 5

.ie 289 283 3 4

.gr 284 275 6 4

.dk 262 254 5 2

.bg 248 238 4 6

.ec 226 217 2 7

.no 223 217 4 2

.za 220 210 5 6

.gt 211 202 2 6

.pk 202 184 4 13

.cz 199 191 6 2

.tw 196 184 3 9

.hr 177 171 3 2

.eg 156 141 4 12

.sk 133 124 5 4

.uy 124 119 2 3

.cr 124 119 2 2

.sa 122 112 3 7

.kr 119 79 24 16

.sv 112 108 1 3

.do 111 107 1 3

.lt 111 107 2 1

.ma 103 93 2 7

.ae 102 99 2 1

.si 100 96 2 2

.bo 86 80 1 5

.pr 73 71 1 1

.lv 61 59 1 1

.py 61 58 1 1

.ee 59 57 1 1

.ba 58 56 2 1

.hn 57 54 1 2

.lk 56 53 2 2

.ke 46 44 1 1

.ng 44 42 1 2

.jo 35 33 1 1

.ni 35 34 1 1

.by 34 34 1 1

.kz 30 28 1 1

.tt 30 29 1 1

.kw 29 28 1 1

.rs 28 27 1 1

.jm 25 24 1 1

.lu 24 24 1 1

.qa 24 23 1 1

.md 23 22 1 1

.lb 20 19 1 1

.mt 19 19 1 1

.bh 19 18 1 1

.is 18 17 1 1

.np 18 17 1 1

.ge 17 16 1 1

.mu 17 16 1 1

.gh 15 14 1 1

.om 13 12 1 1

.pa 13 12 1 1

.ps 12 10 1 1

.az 12 11 1 1

.ug 10 9 1 1

.ly 10 9 1 1

.org 10 2 8 1

.ci 9 8 1 1

.mv 9 8 1 1

.am 9 9 1 1

.cm 9 9 1 1

.cu 9 9 1 1

.bn 9 9 1 1

.sn 8 8 1 1

.tz 7 6 1 1

.et 7 6 1 1

.fj 6 6 1

.bs 6 6 1 1

.cat 6 5 1 1

.bd 5 4 1 1

.mn 5 5 1 1

.mz 5 4 1 1

.net 5 1 4 1

.zw 5 5 1 1

.na 4 4 1 1

.dz 4 3 1

.bw 4 4 1

.mg 4 4 1 1

.kh 4 4 1 1

.dm 3 3 1 1

.cn 3 2 1 1

.vi 3 2 1 1

.bz 3 3 1 1

.zm 3 3 1 1

.rw 2 2 1 1

.af 2 2 1 1

.kg 2 2 1

.cd 2 2 1

.ag 2 2

.uz 2 2 1

.ht 2 2 1 1

.gy 2 2 1

.vc 2 2 1

.gp 2 2 1

.biz 1 1 1

.gl 1 1 1

.bf 1 1 1

.la 1 1 1

.gi 1 1

.ga 1 1 1

.me 1 1 1

.vg 1 1 1

.li 1 1 1 1

.cg 1 1 1 1

.as 1 1

.sc 1 1 1

.vu 1 1 1 1

.ml 1 1

.bj 1 1 1

.gm 1 1 1

.mk 1 1 1 1

.dj 1 1 1

.gg 1 1 1

.ne 1 1 1 1

.ki 1 1

.nu 1 1 1

.st 1 1 1

.ck 1 1

.ao 1 1

.ls 1 1 1

.tm 1 1

.pn 1 1

.sb 1 1

.ai 1 1

.gs 1 1

.ir 1 1 1

.td 1 1

.sl 1 1

.us 1 1

.tv 1 1

.bi 1 1 1

.tk 1 1

.nr 1 1

.edu 1 1 1

.nf 1 1

.mw 1 1

.to 1 1 1

.eu 1 1

.ac 1 1

.je 1 1

.cc 1 1

.sh 1 1

.tl 1 1

.tj 1 1

.cf 1 1

.im 1 1 1

.pro 1 1

.fm 1 1 1 1

.ws 1 1 1

.ad 1 1

.tg 1 1

.sm 1 1

undefined 15,903 13,153 589 2,160

Total 108,691 101,507 2,794 4,368

In alphabetical order

Requests originating from a Google ip address

Service Total Pages Images Other

Wireless 3,029 2,997 31 2

Web search 9 8 1 1

Translate 557 557 1

Toolbar 1 1

Other 362 185 23 154

KeywordTool 1 1

GoogleBot 8,305 7,185 392 728

FeedFetcher 287 92 27 168

Desktop 1 1

Total 12,549 11,024 473 1,053

Requests originating from elsewhere

Service Total Pages Images Other

Wireless 35 33 1 1

Web search 92,116 87,892 2,032 2,192

Translate 25 3 15 6

Toolbar 235 85 37 113

Other 198 73 25 99

Maps 61 27 33 1

Mail 11 3 7 1

Image search 64 26 38 1

GoogleBot? 2,438 2,281 13 144

FeedFetcher 1 1 1

Earth 154 26 127 1

Desktop 810 41 1 769

Total 96,147 90,490 2,328 3,326

Top level domains

.ac 1 1

.ad 1 1

.ae 102 99 2 1

.af 2 2 1 1

.ag 2 2

.ai 1 1

.am 9 9 1 1

.ao 1 1

.ar 1,158 1,113 16 30

.as 1 1

.at 572 553 10 9

.au 2,014 1,965 29 20

.az 12 11 1 1

.ba 58 56 2 1

.bd 5 4 1 1

.be 598 581 9 9

.bf 1 1 1

.bg 248 238 4 6

.bh 19 18 1 1

.bi 1 1 1

.biz 1 1 1

.bj 1 1 1

.bn 9 9 1 1

.bo 86 80 1 5

.br 3,267 3,171 38 58

.bs 6 6 1 1

.bw 4 4 1

.by 34 34 1 1

.bz 3 3 1 1

.ca 3,031 2,942 41 47

.cat 6 5 1 1

.cc 1 1

.cd 2 2 1

.cf 1 1

.cg 1 1 1 1

.ch 602 584 10 8

.ci 9 8 1 1

.ck 1 1

.cl 748 723 11 15

.cm 9 9 1 1

.cn 3 2 1 1

.co 1,361 1,320 12 29

.com 27,844 26,767 593 484

.cr 124 119 2 2

.cu 9 9 1 1

.cz 199 191 6 2

.de 7,170 6,981 112 77

.dj 1 1 1

.dk 262 254 5 2

.dm 3 3 1 1

.do 111 107 1 3

.dz 4 3 1

.ec 226 217 2 7

.edu 1 1 1

.ee 59 57 1 1

.eg 156 141 4 12

.es 2,101 2,020 33 49

.et 7 6 1 1

.eu 1 1

.fi 525 517 5 4

.fj 6 6 1

.fm 1 1 1 1

.fr 3,628 3,511 48 69

.ga 1 1 1

.ge 17 16 1 1

.gg 1 1 1

.gh 15 14 1 1

.gi 1 1

.gl 1 1 1

.gm 1 1 1

.gp 2 2 1

.gr 284 275 6 4

.gs 1 1

.gt 211 202 2 6

.gy 2 2 1

.hk 418 359 27 33

.hn 57 54 1 2

.hr 177 171 3 2

.ht 2 2 1 1

.hu 403 386 11 6

.id 784 570 205 8

.ie 289 283 3 4

.il 393 375 6 11

.im 1 1 1

.in 2,262 2,091 46 126

.ir 1 1 1

.is 18 17 1 1

.it 3,359 3,265 44 50

.je 1 1

.jm 25 24 1 1

.jo 35 33 1 1

.jp 6,016 5,414 91 511

.ke 46 44 1 1

.kg 2 2 1

.kh 4 4 1 1

.ki 1 1

.kr 119 79 24 16

.kw 29 28 1 1

.kz 30 28 1 1

.la 1 1 1

.lb 20 19 1 1

.li 1 1 1 1

.lk 56 53 2 2

.ls 1 1 1

.lt 111 107 2 1

.lu 24 24 1 1

.lv 61 59 1 1

.ly 10 9 1 1

.ma 103 93 2 7

.md 23 22 1 1

.me 1 1 1

.mg 4 4 1 1

.mk 1 1 1 1

.ml 1 1

.mn 5 5 1 1

.mt 19 19 1 1

.mu 17 16 1 1

.mv 9 8 1 1

.mw 1 1

.mx 1,855 1,786 28 41

.my 553 460 85 8

.mz 5 4 1 1

.na 4 4 1 1

.ne 1 1 1 1

.net 5 1 4 1

.nf 1 1

.ng 44 42 1 2

.ni 35 34 1 1

.nl 1,049 1,018 16 15

.no 223 217 4 2

.np 18 17 1 1

.nr 1 1

.nu 1 1 1

.nz 366 336 6 24

.om 13 12 1 1

.org 10 2 8 1

.pa 13 12 1 1

.pe 786 721 10 54

.ph 1,132 1,067 46 18

.pk 202 184 4 13

.pl 2,338 2,283 30 25

.pn 1 1

.pr 73 71 1 1

.pro 1 1

.ps 12 10 1 1

.pt 416 405 6 5

.py 61 58 1 1

.qa 24 23 1 1

.ro 382 359 11 12

.rs 28 27 1 1

.ru 902 805 85 13

.rw 2 2 1 1

.sa 122 112 3 7

.sb 1 1

.sc 1 1 1

.se 634 619 10 5

.sg 399 368 27 4

.sh 1 1

.si 100 96 2 2

.sk 133 124 5 4

.sl 1 1

.sm 1 1

.sn 8 8 1 1

.st 1 1 1

.sv 112 108 1 3

.td 1 1

.tg 1 1

.th 843 649 150 45

.tj 1 1

.tk 1 1

.tl 1 1

.tm 1 1

.to 1 1 1

.tr 757 714 17 26

.tt 30 29 1 1

.tv 1 1

.tw 196 184 3 9

.tz 7 6 1 1

.ua 350 340 4 5

.ug 10 9 1 1

.uk 5,230 5,098 73 60

.us 1 1

.uy 124 119 2 3

.uz 2 2 1

.vc 2 2 1

.ve 608 587 6 15

.vg 1 1 1

.vi 3 2 1 1

.vn 447 344 73 30

.vu 1 1 1 1

.ws 1 1 1

.za 220 210 5 6

.zm 3 3 1 1

.zw 5 5 1 1

undefined 15,903 13,153 589 2,160

Total 108,691 101,507 2,794 4,368

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges
Agents: as for genuine agent strings: too many crawlers indentify themselves as 'GoogleBot' to take this at face value. They are accepted as genuine Google crawler requests only when the ip address matches a known range (see above). Other records that mention GoogleBot are counted as GoogleBot? (question mark, as this may include partners, like DoCoMo). However when the agent string mentions Google Desktop or Google Earth this is always accepted
Service: the service name is based on the agent string (plus for GoogleBot check for ip address, see above), if this is inconclusive it is based on the referer string.
Here is detailed breakdown per service of indicators that pointed to Google (total ≥ 3)

Service Total Originating from
Google ip address Referer mentions
Google url Agent mentions
Google service

Desktop 810 - - Y
Earth 154 - - Y
FeedFetcher 287 Y - Y
GoogleBot 8,305 Y - Y
GoogleBot? 2,435 - - Y
GoogleBot? 3 - Y Y
Image search 64 - Y -
Mail 11 - Y -
Maps 61 - Y -
Other 173 - - Y
Other 5 - Y -
Other 20 - Y Y
Other 208 Y - -
Other 153 Y - Y
Toolbar 158 - - Y
Toolbar 53 - Y -
Toolbar 24 - Y Y
Translate 23 - Y -
Translate 421 Y - Y
Translate 136 Y Y Y
Web search 92,116 - Y -
Web search 9 Y Y -
Wireless 35 - - Y
Wireless 3,029 Y - Y

Top Level Domain 'undefined': requests with top level domain 'undefined' are nearly all requests from anonymous ip addresses (crawler and other services)
Note: averages below 1 are always rounded up to 1

Generated on Tue, Oct 19, 2010 3:24
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.