Wikimedia Traffic Analysis Report

Wikimedia Traffic Analysis Report - Google requests

Daily averages, based on sample period: 1 Oct 2010 - 31 Oct 2010

This analysis is based on a 1:1000 sampled server log (squids) ⇒ all counts x 1000.
See also: Requests by destination or by origin / Methods / Scripts / Skins / Crawlers / Op.Sys. / Browsers / Google

This report shows all requests to Wikimedia servers where a Google server of service was involved in any way, be it the GoogleBot crawler or FeedFetcher collector scripts that run on Google servers, or a user that follows a link from a Google Web or Google Desktop search results page, or from Google Maps or Google Earth etcetera.
Technically speaking three fields in the squid log records are checked for this: client ip address, referer header and user agent header. A request can originate from an ip address which has been registered by Google and/or it can carry a referer tag that tells us a user clicked a link on a Google results page and/or it can carry an agent string that mentions a Google application which can reasonably be assumed to be genuinely Google's. See bottom of page for further details.
In total Google was somehow involved in 43.6% of daily external page^* requests
Google referred to our sites, through its services including search, maps, and Google Earth, 150,685,000 page views per day, representing 40.2% of our external page requests.
Including all of its different search crawlers and services hosted on its servers, Google itself requested another 12,481,000 page pages per day, representing 3.3% of our external page requests.
* = mime type text/html only

In order of request volume

Requests originating from a Google ip address

Service Total Pages Images Other

GoogleBot 8,285 7,203 477 605

Wireless 3,981 3,917 59 5

Translate 966 966 1 1

Other 621 294 41 286

FeedFetcher 321 90 12 219

Web search 12 11 1 1

Toolbar 1 1 1

Desktop 1 1

Maps 1 1

Total 14,186 12,481 589 1,116

Requests originating from elsewhere

Service Total Pages Images Other

Web search 153,315 147,153 3,230 2,931

GoogleBot? 3,300 3,140 32 128

Desktop 999 45 1 954

Other 341 101 104 135

Toolbar 293 119 34 140

Image search 116 56 59 1

Earth 90 27 63 1

Maps 75 33 40 1

Translate 28 5 18 5

Mail 12 4 7 1

Wireless 6 2 3 1

FeedFetcher 1 1 1

Total 158,576 150,685 3,590 4,295

Top level domains

.com 44,866 42,643 1,569 654

.de 11,229 10,969 162 98

.uk 8,437 8,214 148 75

.jp 7,707 6,979 66 662

.fr 7,181 6,998 87 96

.it 5,480 5,327 83 70

.mx 5,461 5,308 63 90

.br 5,447 5,309 66 73

.ca 5,102 4,966 82 55

.pl 4,802 4,696 69 37

.es 4,402 4,252 72 77

.in 3,162 2,944 68 150

.au 2,886 2,822 46 19

.co 2,117 2,069 20 27

.nl 1,972 1,909 44 18

.ru 1,773 1,669 83 20

.ar 1,682 1,622 26 33

.ve 1,546 1,507 11 28

.tr 1,493 1,415 35 43

.se 1,312 1,283 20 9

.cl 1,178 1,144 18 16

.be 1,157 1,119 23 15

.ph 1,119 1,085 22 13

.pe 1,115 1,038 14 63

.id 1,095 1,026 55 13

.ch 1,061 1,030 20 11

.at 1,021 995 15 10

.fi 931 917 9 6

.ua 804 783 9 12

.pt 785 761 15 9

.hu 742 708 23 10

.th 731 607 94 30

.my 695 652 35 7

.hk 688 603 28 57

.ro 678 636 19 23

.vn 669 574 50 44

.il 626 600 10 16

.dk 567 551 12 4

.gr 519 502 11 7

.no 512 498 11 3

.sg 509 491 13 5

.ec 499 484 4 11

.nz 468 445 7 16

.ie 454 443 6 5

.bg 452 432 9 11

.cz 418 400 14 4

.tw 359 339 7 13

.hr 353 337 9 6

.do 352 342 2 8

.eg 327 298 7 22

.pk 326 301 8 16

.za 315 301 8 5

.sk 290 274 12 5

.ma 271 249 3 19

.lt 262 251 7 4

.sa 218 203 4 11

.ae 186 180 4 2

.cr 183 177 3 2

.si 181 174 5 2

.uy 177 173 2 2

.pr 177 174 3 1

.gt 172 166 2 4

.bo 171 160 2 8

.kr 171 131 21 19

.sv 165 160 2 3

.lv 135 130 4 1

.ee 126 123 2 1

.ba 115 109 4 3

.hn 94 91 1 2

.py 93 90 1 1

.lk 79 75 3 2

.by 78 77 1 1

.jo 78 74 2 3

.kz 72 70 1 1

.ke 68 66 1 2

.ng 63 60 1 2

.rs 61 57 2 2

.jm 56 55 1 1

.md 56 55 1 1

.tt 54 53 1 1

.kw 49 47 1 1

.qa 46 44 1 1

.ge 44 41 2 1

.lu 43 41 1 1

.ps 40 37 1 2

.is 38 37 1 1

.bh 37 36 1 1

.lb 37 35 1 1

.mt 36 34 1 1

.om 30 28 1 2

.gh 30 29 1 1

.ni 29 28 1 1

.mu 28 27 1 1

.az 26 24 1 1

.ug 23 22 1 1

.ly 23 20 1 2

.cu 22 21 1 1

.pa 21 20 1 1

.am 20 19 1 1

.org 20 2 18 1

.np 19 18 1 1

.cm 18 17 1 1

.ci 15 14 1 1

.mn 15 14 1 1

.cat 13 13 1 1

.et 13 12 1 1

.bs 12 11 1 1

.bn 12 12 1 1

.mv 11 11 1 1

.sn 11 11 1 1

.mz 11 11 1 1

.zw 11 11 1 1

.dz 10 8 1 1

.net 10 1 8 1

.tz 9 9 1 1

.mg 9 9 1 1

.fj 8 8 1 1

.bw 8 7 1 1

.na 7 7 1 1

.bz 7 6 1 1

.gy 7 7 1 1

.dm 5 5 1

.kg 5 5 1 1

.ag 5 4 1

.kh 5 5 1 1

.uz 5 5 1 1

.gp 5 4 1

.bd 4 4 1 1

.rw 4 4 1 1

.af 4 3 1 1

.cd 4 3 1 1

.ht 4 4 1 1

.zm 4 4 1 1

.vc 4 4 1 1

.cn 3 3 1 1

.vi 3 3 1

.mk 3 2 1 1

.bf 2 2 1

.ls 2 2 1

.ga 2 2 1 1

.gi 2 2 1

.me 2 2 1 1

.vg 2 2 1 1

.li 2 2 1

.as 2 2 1 1

.ml 2 2 1 1

.ne 1 1

.biz 1 1 1

.gl 1 1

.la 1 1 1 1

.sb 1 1 1 1

.ai 1 1 1

.cg 1 1 1

.sl 1 1

.us 1 1 1

.bi 1 1 1

.sc 1 1

.bj 1 1 1 1

.mw 1 1 1

.gm 1 1 1

.to 1 1 1

.tl 1 1 1

.fm 1 1 1 1

.ws 1 1 1 1

.ad 1 1

.tg 1 1 1

.sm 1 1 1

.dj 1 1 1

.gg 1 1 1 1

.ki 1 1

.nu 1 1

.st 1 1

.ck 1 1

.tc 1 1

.ao 1 1

.tm 1 1

.ms 1 1

.pn 1 1

.ir 1 1 1

.tv 1 1

.tk 1 1

.su 1 1

.nr 1 1

.vu 1 1

.edu 1 1

.nf 1 1

.eu 1 1 1

.ac 1 1

.je 1 1 1

.cc 1 1

.sh 1 1

.tj 1 1

.cf 1 1

.im 1 1

.pro 1 1

undefined 18,343 15,258 628 2,457

Total 172,762 163,164 4,173 5,408

In alphabetical order

Requests originating from a Google ip address

Service Total Pages Images Other

Wireless 3,981 3,917 59 5

Web search 12 11 1 1

Translate 966 966 1 1

Toolbar 1 1 1

Other 621 294 41 286

Maps 1 1

GoogleBot 8,285 7,203 477 605

FeedFetcher 321 90 12 219

Desktop 1 1

Total 14,186 12,481 589 1,116

Requests originating from elsewhere

Service Total Pages Images Other

Wireless 6 2 3 1

Web search 153,315 147,153 3,230 2,931

Translate 28 5 18 5

Toolbar 293 119 34 140

Other 341 101 104 135

Maps 75 33 40 1

Mail 12 4 7 1

Image search 116 56 59 1

GoogleBot? 3,300 3,140 32 128

FeedFetcher 1 1 1

Earth 90 27 63 1

Desktop 999 45 1 954

Total 158,576 150,685 3,590 4,295

Top level domains

.ac 1 1

.ad 1 1

.ae 186 180 4 2

.af 4 3 1 1

.ag 5 4 1

.ai 1 1 1

.am 20 19 1 1

.ao 1 1

.ar 1,682 1,622 26 33

.as 2 2 1 1

.at 1,021 995 15 10

.au 2,886 2,822 46 19

.az 26 24 1 1

.ba 115 109 4 3

.bd 4 4 1 1

.be 1,157 1,119 23 15

.bf 2 2 1

.bg 452 432 9 11

.bh 37 36 1 1

.bi 1 1 1

.biz 1 1 1

.bj 1 1 1 1

.bn 12 12 1 1

.bo 171 160 2 8

.br 5,447 5,309 66 73

.bs 12 11 1 1

.bw 8 7 1 1

.by 78 77 1 1

.bz 7 6 1 1

.ca 5,102 4,966 82 55

.cat 13 13 1 1

.cc 1 1

.cd 4 3 1 1

.cf 1 1

.cg 1 1 1

.ch 1,061 1,030 20 11

.ci 15 14 1 1

.ck 1 1

.cl 1,178 1,144 18 16

.cm 18 17 1 1

.cn 3 3 1 1

.co 2,117 2,069 20 27

.com 44,866 42,643 1,569 654

.cr 183 177 3 2

.cu 22 21 1 1

.cz 418 400 14 4

.de 11,229 10,969 162 98

.dj 1 1 1

.dk 567 551 12 4

.dm 5 5 1

.do 352 342 2 8

.dz 10 8 1 1

.ec 499 484 4 11

.edu 1 1

.ee 126 123 2 1

.eg 327 298 7 22

.es 4,402 4,252 72 77

.et 13 12 1 1

.eu 1 1 1

.fi 931 917 9 6

.fj 8 8 1 1

.fm 1 1 1 1

.fr 7,181 6,998 87 96

.ga 2 2 1 1

.ge 44 41 2 1

.gg 1 1 1 1

.gh 30 29 1 1

.gi 2 2 1

.gl 1 1

.gm 1 1 1

.gp 5 4 1

.gr 519 502 11 7

.gt 172 166 2 4

.gy 7 7 1 1

.hk 688 603 28 57

.hn 94 91 1 2

.hr 353 337 9 6

.ht 4 4 1 1

.hu 742 708 23 10

.id 1,095 1,026 55 13

.ie 454 443 6 5

.il 626 600 10 16

.im 1 1

.in 3,162 2,944 68 150

.ir 1 1 1

.is 38 37 1 1

.it 5,480 5,327 83 70

.je 1 1 1

.jm 56 55 1 1

.jo 78 74 2 3

.jp 7,707 6,979 66 662

.ke 68 66 1 2

.kg 5 5 1 1

.kh 5 5 1 1

.ki 1 1

.kr 171 131 21 19

.kw 49 47 1 1

.kz 72 70 1 1

.la 1 1 1 1

.lb 37 35 1 1

.li 2 2 1

.lk 79 75 3 2

.ls 2 2 1

.lt 262 251 7 4

.lu 43 41 1 1

.lv 135 130 4 1

.ly 23 20 1 2

.ma 271 249 3 19

.md 56 55 1 1

.me 2 2 1 1

.mg 9 9 1 1

.mk 3 2 1 1

.ml 2 2 1 1

.mn 15 14 1 1

.ms 1 1

.mt 36 34 1 1

.mu 28 27 1 1

.mv 11 11 1 1

.mw 1 1 1

.mx 5,461 5,308 63 90

.my 695 652 35 7

.mz 11 11 1 1

.na 7 7 1 1

.ne 1 1

.net 10 1 8 1

.nf 1 1

.ng 63 60 1 2

.ni 29 28 1 1

.nl 1,972 1,909 44 18

.no 512 498 11 3

.np 19 18 1 1

.nr 1 1

.nu 1 1

.nz 468 445 7 16

.om 30 28 1 2

.org 20 2 18 1

.pa 21 20 1 1

.pe 1,115 1,038 14 63

.ph 1,119 1,085 22 13

.pk 326 301 8 16

.pl 4,802 4,696 69 37

.pn 1 1

.pr 177 174 3 1

.pro 1 1

.ps 40 37 1 2

.pt 785 761 15 9

.py 93 90 1 1

.qa 46 44 1 1

.ro 678 636 19 23

.rs 61 57 2 2

.ru 1,773 1,669 83 20

.rw 4 4 1 1

.sa 218 203 4 11

.sb 1 1 1 1

.sc 1 1

.se 1,312 1,283 20 9

.sg 509 491 13 5

.sh 1 1

.si 181 174 5 2

.sk 290 274 12 5

.sl 1 1

.sm 1 1 1

.sn 11 11 1 1

.st 1 1

.su 1 1

.sv 165 160 2 3

.tc 1 1

.tg 1 1 1

.th 731 607 94 30

.tj 1 1

.tk 1 1

.tl 1 1 1

.tm 1 1

.to 1 1 1

.tr 1,493 1,415 35 43

.tt 54 53 1 1

.tv 1 1

.tw 359 339 7 13

.tz 9 9 1 1

.ua 804 783 9 12

.ug 23 22 1 1

.uk 8,437 8,214 148 75

.us 1 1 1

.uy 177 173 2 2

.uz 5 5 1 1

.vc 4 4 1 1

.ve 1,546 1,507 11 28

.vg 2 2 1 1

.vi 3 3 1

.vn 669 574 50 44

.vu 1 1

.ws 1 1 1 1

.za 315 301 8 5

.zm 4 4 1 1

.zw 11 11 1 1

undefined 18,343 15,258 628 2,457

Total 172,762 163,164 4,173 5,408

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges
Agents: as for genuine agent strings: too many crawlers indentify themselves as 'GoogleBot' to take this at face value. They are accepted as genuine Google crawler requests only when the ip address matches a known range (see above). Other records that mention GoogleBot are counted as GoogleBot? (question mark, as this may include partners, like DoCoMo). However when the agent string mentions Google Desktop or Google Earth this is always accepted
Service: the service name is based on the agent string (plus for GoogleBot check for ip address, see above), if this is inconclusive it is based on the referer string.
Here is detailed breakdown per service of indicators that pointed to Google (total ≥ 3)

Service Total Originating from
Google ip address Referer mentions
Google url Agent mentions
Google service

Desktop 999 - - Y
Earth 90 - - Y
FeedFetcher 321 Y - Y
GoogleBot 8,285 Y - Y
GoogleBot? 3,293 - - Y
GoogleBot? 7 - Y Y
Image search 116 - Y -
Mail 12 - Y -
Maps 75 - Y -
Other 236 - - Y
Other 9 - Y -
Other 96 - Y Y
Other 312 Y - -
Other 308 Y - Y
Toolbar 201 - - Y
Toolbar 60 - Y -
Toolbar 32 - Y Y
Translate 27 - Y -
Translate 732 Y - Y
Translate 235 Y Y Y
Web search 153,315 - Y -
Web search 12 Y Y -
Wireless 6 - - Y
Wireless 3,981 Y - Y

Top Level Domain 'undefined': requests with top level domain 'undefined' are nearly all requests from anonymous ip addresses (crawler and other services)
Note: averages below 1 are always rounded up to 1

Generated on Tue, Jan 11, 2011 6:23
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.