Wikimedia Traffic Analysis Report

Wikimedia Traffic Analysis Report - Google requests

Daily averages, based on sample period: 1 Mar 2011 - 31 Mar 2011

This analysis is based on a 1:1000 sampled server log (squids) ⇒ all counts x 1000.
See also: Requests by destination or by origin / Methods / Scripts / Skins / Crawlers / Op.Sys. / Browsers / Google

This report shows all requests to Wikimedia servers where a Google server of service was involved in any way, be it the GoogleBot crawler or FeedFetcher collector scripts that run on Google servers, or a user that follows a link from a Google Web or Google Desktop search results page, or from Google Maps or Google Earth etcetera.
Technically speaking three fields in the squid log records are checked for this: client ip address, referer header and user agent header. A request can originate from an ip address which has been registered by Google and/or it can carry a referer tag that tells us a user clicked a link on a Google results page and/or it can carry an agent string that mentions a Google application which can reasonably be assumed to be genuinely Google's. See bottom of page for further details.
In total Google was somehow involved in 42.3% of daily external page^* requests
Google referred to our sites, through its services including search, maps, and Google Earth, 148,273,000 page views per day, representing 37.3% of our external page requests.
Including all of its different search crawlers and services hosted on its servers, Google itself requested another 19,839,000 page pages per day, representing 5.0% of our external page requests.
* = mime type text/html only

In order of request volume

Requests originating from a Google ip address

Service Total Pages Images Other

GoogleBot 15,774 14,163 595 1,016

Wireless 3,943 3,879 59 5

Other 1,568 637 432 500

Translate 1,049 1,049 1

FeedFetcher 329 100 19 210

Web search 11 11 1

Image search 1 1

Toolbar 1 1

Desktop 1 1

Maps 1 1

Total 22,674 19,839 1,105 1,732

Requests originating from elsewhere

Service Total Pages Images Other

Web search 147,140 143,228 2,292 1,620

GoogleBot? 5,068 4,652 64 352

Desktop 962 67 895

Other 453 154 130 168

Maps 320 28 290 1

Toolbar 213 78 27 107

Earth 91 28 63 1

Image search 62 27 34 1

Translate 24 6 15 3

Mail 11 3 8 1

Wireless 2 1 1 1

FeedFetcher 1 1 1

Total 154,347 148,273 2,924 3,147

Top level domains

.com 51,180 49,444 1,352 385

.de 9,447 9,311 94 42

.uk 8,583 8,391 153 40

.jp 7,054 6,347 41 666

.fr 6,763 6,631 95 37

.it 5,636 5,516 91 29

.br 5,303 5,215 56 32

.ca 4,825 4,735 63 27

.es 4,429 4,324 74 31

.pl 4,085 3,987 84 15

.au 2,841 2,787 42 12

.mx 2,831 2,784 28 18

.in 2,754 2,633 54 67

.ru 2,153 2,054 90 10

.nl 1,966 1,919 40 7

.tr 1,534 1,486 31 17

.se 1,248 1,227 19 3

.co 1,191 1,174 11 6

.be 1,123 1,096 21 6

.ar 1,053 1,031 14 8

.ch 1,043 1,021 18 4

.at 991 970 16 5

.ph 975 947 21 7

.ua 923 905 12 6

.fi 821 806 12 2

.id 800 752 40 8

.pt 761 746 12 3

.cl 726 711 10 5

.hk 701 656 23 23

.pe 677 659 7 11

.hu 673 648 21 4

.th 672 605 56 11

.ro 637 615 15 7

.my 623 596 24 4

.ve 610 600 6 4

.vn 590 532 37 22

.il 573 557 10 6

.dk 547 535 11 1

.no 540 529 10 1

.gr 503 490 10 4

.sg 479 467 8 3

.ie 462 454 5 3

.cz 453 436 15 2

.nz 444 436 7 1

.bg 418 406 9 4

.tw 388 372 10 6

.hr 347 335 9 3

.za 333 322 7 4

.sk 297 285 10 2

.pk 283 269 7 7

.lt 274 266 7 1

.eg 250 234 8 8

.sa 235 225 7 3

.ma 215 207 3 5

.org 187 4 182 1

.si 177 171 5 1

.ae 172 168 3 1

.kr 172 145 15 11

.do 152 150 1 1

.pr 139 137 2 1

.ec 137 134 2 1

.gt 135 133 1 1

.ee 120 117 2 1

.ba 119 113 4 1

.lv 119 116 3 1

.uy 96 94 1 1

.cr 93 91 2 1

.by 92 90 1 1

.bo 84 80 1 3

.sv 83 82 1 1

.kz 77 75 1 1

.jo 74 71 2 1

.ng 71 68 1 2

.lk 66 62 3 1

.rs 66 63 2 1

.ke 62 60 1 1

.md 61 60 1 1

.ge 55 53 2 1

.jm 49 48 1 1

.tt 48 48 1 1

.hn 45 44 1 1

.py 44 44 1 1

.qa 43 41 1 1

.kw 42 41 1 1

.lb 41 38 2 1

.lu 39 38 1 1

.is 36 36 1 1

.ps 33 31 1 1

.mt 32 31 1 1

.az 30 29 1 1

.om 29 28 1 1

.net 29 4 23 1

.ni 28 27 1 1

.bh 27 27 1 1

.mu 27 26 1 1

.am 26 26 1 1

.gh 26 25 1 1

.pa 21 21 1 1

.np 19 18 1 1

.ug 18 17 1 1

.cm 17 16 1 1

.mz 16 16 1 1

.mn 15 14 1 1

.cat 15 14 1 1

.sn 14 13 1 1

.et 14 14 1 1

.tz 13 12 1 1

.zw 12 12 1 1

.ci 10 10 1 1

.bn 10 10 1 1

.mv 9 9 1 1

.na 8 7 1 1

.bw 8 8 1 1

.cu 8 8 1 1

.gp 8 8 1 1

.dz 7 6 1 1

.kh 7 7 1 1

.bs 7 7 1 1

.kg 6 6 1 1

.mg 6 6 1 1

.bz 6 6 1 1

.uz 6 6 1 1

.gy 6 5 1

.fj 5 5 1 1

.cd 5 4 1 1

.zm 5 5 1

.rw 4 4 1

.ag 4 4 1 1

.ht 4 4 1

.eu 4 1 4 1

.bd 3 3 1 1

.tn 3 3 1

.dm 3 3

.vi 3 3 1 1

.ml 3 3 1 1

.cc 3 1 3 1

.mk 3 3 1 1

.vc 3 3 1 1

.biz 2 1 1 1

.bf 2 2 1

.ao 2 2 1 1

.af 2 2 1 1

.la 2 2 1

.gi 2 2

.ga 2 2

.ly 2 2 1

.me 2 2 1 1

.li 2 2 1

.us 2 1 2

.sc 2 1 1

.bj 2 2 1

.edu 2 1 1 1

.ne 1 1

.gl 1 1

.ls 1 1 1 1

.tm 1 1

.iq 1 1 1 1

.ai 1 1

.vg 1 1 1 1

.ir 1 1 1

.sl 1 1

.bi 1 1 1

.as 1 1 1

.tk 1 1 1

.cn 1 1 1

.gm 1 1

.mw 1 1 1 1

.to 1 1 1

.tl 1 1 1

.ws 1 1 1 1

.ad 1 1 1

.tg 1 1 1

.sm 1 1 1

.dj 1 1 1

.gg 1 1 1

.ki 1 1

.nu 1 1 1

.st 1 1

.hm 1 1

.ck 1 1 1 1

.ms 1 1 1

.pn 1 1

.sb 1 1

.td 1 1

.cg 1 1 1

.tv 1 1 1

.su 1 1

.nr 1 1 1

.vu 1 1

.nf 1 1

.ac 1 1

.gov 1 1

.je 1 1 1

.sh 1 1

.cf 1 1

.tj 1 1

.im 1 1 1

.fm 1 1 1

undefined 28,078 24,091 800 3,187

Total 177,015 168,108 4,026 4,875

In alphabetical order

Requests originating from a Google ip address

Service Total Pages Images Other

Wireless 3,943 3,879 59 5

Web search 11 11 1

Translate 1,049 1,049 1

Toolbar 1 1

Other 1,568 637 432 500

Maps 1 1

Image search 1 1

GoogleBot 15,774 14,163 595 1,016

FeedFetcher 329 100 19 210

Desktop 1 1

Total 22,674 19,839 1,105 1,732

Requests originating from elsewhere

Service Total Pages Images Other

Wireless 2 1 1 1

Web search 147,140 143,228 2,292 1,620

Translate 24 6 15 3

Toolbar 213 78 27 107

Other 453 154 130 168

Maps 320 28 290 1

Mail 11 3 8 1

Image search 62 27 34 1

GoogleBot? 5,068 4,652 64 352

FeedFetcher 1 1 1

Earth 91 28 63 1

Desktop 962 67 895

Total 154,347 148,273 2,924 3,147

Top level domains

.ac 1 1

.ad 1 1 1

.ae 172 168 3 1

.af 2 2 1 1

.ag 4 4 1 1

.ai 1 1

.am 26 26 1 1

.ao 2 2 1 1

.ar 1,053 1,031 14 8

.as 1 1 1

.at 991 970 16 5

.au 2,841 2,787 42 12

.az 30 29 1 1

.ba 119 113 4 1

.bd 3 3 1 1

.be 1,123 1,096 21 6

.bf 2 2 1

.bg 418 406 9 4

.bh 27 27 1 1

.bi 1 1 1

.biz 2 1 1 1

.bj 2 2 1

.bn 10 10 1 1

.bo 84 80 1 3

.br 5,303 5,215 56 32

.bs 7 7 1 1

.bw 8 8 1 1

.by 92 90 1 1

.bz 6 6 1 1

.ca 4,825 4,735 63 27

.cat 15 14 1 1

.cc 3 1 3 1

.cd 5 4 1 1

.cf 1 1

.cg 1 1 1

.ch 1,043 1,021 18 4

.ci 10 10 1 1

.ck 1 1 1 1

.cl 726 711 10 5

.cm 17 16 1 1

.cn 1 1 1

.co 1,191 1,174 11 6

.com 51,180 49,444 1,352 385

.cr 93 91 2 1

.cu 8 8 1 1

.cz 453 436 15 2

.de 9,447 9,311 94 42

.dj 1 1 1

.dk 547 535 11 1

.dm 3 3

.do 152 150 1 1

.dz 7 6 1 1

.ec 137 134 2 1

.edu 2 1 1 1

.ee 120 117 2 1

.eg 250 234 8 8

.es 4,429 4,324 74 31

.et 14 14 1 1

.eu 4 1 4 1

.fi 821 806 12 2

.fj 5 5 1 1

.fm 1 1 1

.fr 6,763 6,631 95 37

.ga 2 2

.ge 55 53 2 1

.gg 1 1 1

.gh 26 25 1 1

.gi 2 2

.gl 1 1

.gm 1 1

.gov 1 1

.gp 8 8 1 1

.gr 503 490 10 4

.gt 135 133 1 1

.gy 6 5 1

.hk 701 656 23 23

.hm 1 1

.hn 45 44 1 1

.hr 347 335 9 3

.ht 4 4 1

.hu 673 648 21 4

.id 800 752 40 8

.ie 462 454 5 3

.il 573 557 10 6

.im 1 1 1

.in 2,754 2,633 54 67

.iq 1 1 1 1

.ir 1 1 1

.is 36 36 1 1

.it 5,636 5,516 91 29

.je 1 1 1

.jm 49 48 1 1

.jo 74 71 2 1

.jp 7,054 6,347 41 666

.ke 62 60 1 1

.kg 6 6 1 1

.kh 7 7 1 1

.ki 1 1

.kr 172 145 15 11

.kw 42 41 1 1

.kz 77 75 1 1

.la 2 2 1

.lb 41 38 2 1

.li 2 2 1

.lk 66 62 3 1

.ls 1 1 1 1

.lt 274 266 7 1

.lu 39 38 1 1

.lv 119 116 3 1

.ly 2 2 1

.ma 215 207 3 5

.md 61 60 1 1

.me 2 2 1 1

.mg 6 6 1 1

.mk 3 3 1 1

.ml 3 3 1 1

.mn 15 14 1 1

.ms 1 1 1

.mt 32 31 1 1

.mu 27 26 1 1

.mv 9 9 1 1

.mw 1 1 1 1

.mx 2,831 2,784 28 18

.my 623 596 24 4

.mz 16 16 1 1

.na 8 7 1 1

.ne 1 1

.net 29 4 23 1

.nf 1 1

.ng 71 68 1 2

.ni 28 27 1 1

.nl 1,966 1,919 40 7

.no 540 529 10 1

.np 19 18 1 1

.nr 1 1 1

.nu 1 1 1

.nz 444 436 7 1

.om 29 28 1 1

.org 187 4 182 1

.pa 21 21 1 1

.pe 677 659 7 11

.ph 975 947 21 7

.pk 283 269 7 7

.pl 4,085 3,987 84 15

.pn 1 1

.pr 139 137 2 1

.ps 33 31 1 1

.pt 761 746 12 3

.py 44 44 1 1

.qa 43 41 1 1

.ro 637 615 15 7

.rs 66 63 2 1

.ru 2,153 2,054 90 10

.rw 4 4 1

.sa 235 225 7 3

.sb 1 1

.sc 2 1 1

.se 1,248 1,227 19 3

.sg 479 467 8 3

.sh 1 1

.si 177 171 5 1

.sk 297 285 10 2

.sl 1 1

.sm 1 1 1

.sn 14 13 1 1

.st 1 1

.su 1 1

.sv 83 82 1 1

.td 1 1

.tg 1 1 1

.th 672 605 56 11

.tj 1 1

.tk 1 1 1

.tl 1 1 1

.tm 1 1

.tn 3 3 1

.to 1 1 1

.tr 1,534 1,486 31 17

.tt 48 48 1 1

.tv 1 1 1

.tw 388 372 10 6

.tz 13 12 1 1

.ua 923 905 12 6

.ug 18 17 1 1

.uk 8,583 8,391 153 40

.us 2 1 2

.uy 96 94 1 1

.uz 6 6 1 1

.vc 3 3 1 1

.ve 610 600 6 4

.vg 1 1 1 1

.vi 3 3 1 1

.vn 590 532 37 22

.vu 1 1

.ws 1 1 1 1

.za 333 322 7 4

.zm 5 5 1

.zw 12 12 1 1

undefined 28,078 24,091 800 3,187

Total 177,015 168,108 4,026 4,875

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges
Agents: as for genuine agent strings: too many crawlers indentify themselves as 'GoogleBot' to take this at face value. They are accepted as genuine Google crawler requests only when the ip address matches a known range (see above). Other records that mention GoogleBot are counted as GoogleBot? (question mark, as this may include partners, like DoCoMo). However when the agent string mentions Google Desktop or Google Earth this is always accepted
Service: the service name is based on the agent string (plus for GoogleBot check for ip address, see above), if this is inconclusive it is based on the referer string.
Here is detailed breakdown per service of indicators that pointed to Google (total ≥ 3)

Service Total Originating from
Google ip address Referer mentions
Google url Agent mentions
Google service

Desktop 962 - - Y
Earth 91 - - Y
FeedFetcher 329 Y - Y
GoogleBot 15,774 Y - Y
GoogleBot? 5,023 - - Y
GoogleBot? 45 - Y Y
Image search 62 - Y -
Mail 11 - Y -
Maps 320 - Y -
Other 298 - - Y
Other 25 - Y -
Other 130 - Y Y
Other 227 Y - -
Other 1,338 Y - Y
Other 3 Y Y Y
Toolbar 138 - - Y
Toolbar 49 - Y -
Toolbar 26 - Y Y
Translate 22 - Y -
Translate 832 Y - Y
Translate 217 Y Y Y
Web search 147,140 - Y -
Web search 11 Y Y -
Wireless 3,943 Y - Y

Top Level Domain 'undefined': requests with top level domain 'undefined' are nearly all requests from anonymous ip addresses (crawler and other services)
Note: averages below 1 are always rounded up to 1

Generated on Wed, Apr 20, 2011 15:09
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.