Wikimedia Traffic Analysis Report

Wikimedia Traffic Analysis Report - Google requests

Daily averages, based on sample period: 1 May 2011 - 31 May 2011

This analysis is based on a 1:1000 sampled server log (squids) ⇒ all counts x 1000.
See also: Requests by destination or by origin / Methods / Scripts / Skins / Crawlers / Op.Sys. / Browsers / Google

This report shows all requests to Wikimedia servers where a Google server of service was involved in any way, be it the GoogleBot crawler or FeedFetcher collector scripts that run on Google servers, or a user that follows a link from a Google Web or Google Desktop search results page, or from Google Maps or Google Earth etcetera.
Technically speaking three fields in the squid log records are checked for this: client ip address, referer header and user agent header. A request can originate from an ip address which has been registered by Google and/or it can carry a referer tag that tells us a user clicked a link on a Google results page and/or it can carry an agent string that mentions a Google application which can reasonably be assumed to be genuinely Google's. See bottom of page for further details.
In total Google was somehow involved in 41.8% of daily external page^* requests
Google referred to our sites, through its services including search, maps, and Google Earth, 149,943,000 page views per day, representing 37.6% of our external page requests.
Including all of its different search crawlers and services hosted on its servers, Google itself requested another 16,863,000 page pages per day, representing 4.2% of our external page requests.
* = mime type text/html only

In order of request volume

Requests originating from a Google ip address

Service Total Pages Images Other

GoogleBot 12,529 11,039 692 798

Wireless 3,851 3,775 65 11

Other 2,183 919 425 839

Translate 1,028 1,026 1 1

FeedFetcher 331 92 21 217

Web search 13 12 1 1

Image search 1 1

Toolbar 1 1 1

Maps 1 1 1 1

Total 19,935 16,863 1,204 1,866

Requests originating from elsewhere

Service Total Pages Images Other

Web search 146,890 142,968 2,294 1,629

GoogleBot? 7,251 6,620 86 545

Desktop 852 30 822

Other 516 174 108 234

Maps 178 33 144 1

Toolbar 167 62 21 84

Earth 54 23 31 1

Image search 53 25 27 1

Translate 19 4 12 3

Mail 7 2 5 1

Wireless 2 1 1 1

FeedFetcher 1 1 1

Total 155,990 149,943 2,729 3,319

Top level domains

.com 51,614 49,871 1,204 540

.de 9,360 9,219 100 41

.uk 8,731 8,589 107 34

.jp 7,468 6,832 42 593

.fr 6,526 6,392 102 32

.it 5,505 5,390 87 28

.br 5,415 5,314 68 34

.ca 4,445 4,359 57 29

.es 4,330 4,221 82 27

.pl 3,997 3,906 76 15

.au 2,973 2,924 38 11

.in 2,864 2,752 52 60

.ru 2,251 2,156 85 9

.mx 2,190 2,147 28 15

.nl 1,906 1,862 37 7

.tr 1,448 1,404 30 14

.ar 1,255 1,223 20 12

.se 1,242 1,219 20 3

.be 1,209 1,182 23 5

.ch 1,051 1,025 22 4

.co 989 971 12 6

.at 961 935 21 5

.ua 919 901 11 6

.cl 792 771 15 6

.pt 778 760 14 4

.fi 767 754 11 2

.id 738 694 38 6

.th 721 666 45 9

.hk 706 649 32 25

.ve 677 665 7 5

.pe 664 644 10 11

.ph 655 640 12 4

.il 651 636 10 5

.hu 636 612 21 3

.ro 628 607 16 5

.my 568 546 18 4

.no 543 531 10 2

.dk 535 524 10 2

.vn 527 480 34 13

.cz 512 494 16 2

.sg 498 488 7 3

.nz 495 485 8 2

.ie 490 485 4 2

.gr 482 470 9 3

.tw 408 394 8 6

.bg 389 378 8 3

.za 362 352 7 3

.hr 330 319 9 2

.pk 324 309 8 7

.sk 299 284 13 2

.lt 264 256 7 1

.org 254 5 247 2

.eg 233 219 8 6

.sa 233 223 5 4

.ma 206 198 4 4

.ae 194 189 4 1

.ec 186 182 2 2

.si 175 170 5 1

.kr 164 143 13 9

.do 129 127 1 1

.ee 122 118 3 1

.lv 121 117 3 1

.gt 115 113 1 1

.ba 115 110 4 1

.pr 109 107 2 1

.uy 104 101 2 1

.by 104 103 1 1

.tn 84 82 2 1

.cr 84 82 1 1

.kz 82 80 1 1

.bo 73 70 1 1

.jo 73 71 2 1

.sv 68 66 1 1

.ng 68 66 1 1

.lk 68 64 3 1

.rs 64 61 2 1

.md 54 53 1 1

.tt 53 52 1 1

.ke 50 49 1 1

.ge 50 48 2 1

.jm 46 45 1 1

.py 45 44 1 1

.qa 45 44 1 1

.kw 42 40 1 1

.hn 40 39 1 1

.mt 40 39 1 1

.lb 40 39 1 1

.lu 37 37 1 1

.is 34 33 1 1

.az 33 31 1 1

.bh 30 29 1 1

.ps 29 27 1 1

.mu 29 28 1 1

.gh 28 27 1 1

.net 28 4 22 2

.am 24 23 1 1

.ni 24 24 1 1

.om 23 23 1 1

.np 21 20 1 1

.iq 20 18 1 1

.pa 19 19 1 1

.sn 17 16 1 1

.tz 17 16 1 1

.ug 15 14 1 1

.cm 15 14 1 1

.mn 14 13 1 1

.cat 14 14 1 1

.et 14 13 1 1

.mz 13 13 1 1

.bn 12 11 1 1

.mv 10 10 1 1

.bs 10 10 1 1

.zw 10 10 1 1

.gp 9 9 1 1

.ci 8 8 1 1

.kg 8 7 1 1

.kh 8 7 1 1

.uz 8 8 1 1

.cu 8 8 1 1

.gy 7 6 1 1

.na 6 6 1 1

.mg 6 6 1 1

.bz 6 5 1 1

.dm 5 4 1

.dz 5 5 1 1

.bw 5 5 1 1

.ws 5 1 4 1

.rw 4 4 1

.us 4 2 2 1

.cd 4 4 1

.ag 4 3 1 1

.ht 4 4 1

.eu 4 1 4 1

.zm 4 4 1

.vc 4 4 1

.bd 3 3 1 1

.biz 3 1 2 1

.af 3 3 1 1

.fj 3 3 1

.li 3 3 1

.vi 3 3 1 1

.ml 3 2 1 1

.mk 3 3 1

.bf 2 2 1 1

.ao 2 2 1

.ga 2 2 1

.la 2 2 1

.gi 2 2

.bj 2 1 1

.mw 2 2

.dj 2 2 1

.ne 1 1

.gl 1 1 1

.ls 1 1 1

.tm 1 1

.sb 1 1

.ly 1 1 1 1

.ai 1 1

.me 1 1 1 1

.vg 1 1

.ir 1 1 1 1

.sl 1 1 1

.bi 1 1 1

.as 1 1 1

.sc 1 1

.cn 1 1 1

.gm 1 1 1 1

.to 1 1 1

.cc 1 1 1 1

.tl 1 1 1

.fm 1 1 1

.ad 1 1 1 1

.tg 1 1 1

.gg 1 1 1

.ki 1 1

.nu 1 1 1

.st 1 1 1

.hm 1 1

.ck 1 1 1

.ms 1 1 1

.pn 1 1

.cg 1 1 1

.tv 1 1 1

.gd 1 1

.tk 1 1 1

.su 1 1

.nr 1 1

.vu 1 1

.edu 1 1 1

.ac 1 1 1

.je 1 1

.sh 1 1

.io 1 1

.so 1 1

.tj 1 1

.cf 1 1

.im 1 1

.sm 1 1 1 1

undefined 27,377 23,094 825 3,457

Total 175,926 166,801 3,930 5,176

In alphabetical order

Requests originating from a Google ip address

Service Total Pages Images Other

Wireless 3,851 3,775 65 11

Web search 13 12 1 1

Translate 1,028 1,026 1 1

Toolbar 1 1 1

Other 2,183 919 425 839

Maps 1 1 1 1

Image search 1 1

GoogleBot 12,529 11,039 692 798

FeedFetcher 331 92 21 217

Total 19,935 16,863 1,204 1,866

Requests originating from elsewhere

Service Total Pages Images Other

Wireless 2 1 1 1

Web search 146,890 142,968 2,294 1,629

Translate 19 4 12 3

Toolbar 167 62 21 84

Other 516 174 108 234

Maps 178 33 144 1

Mail 7 2 5 1

Image search 53 25 27 1

GoogleBot? 7,251 6,620 86 545

FeedFetcher 1 1 1

Earth 54 23 31 1

Desktop 852 30 822

Total 155,990 149,943 2,729 3,319

Top level domains

.ac 1 1 1

.ad 1 1 1 1

.ae 194 189 4 1

.af 3 3 1 1

.ag 4 3 1 1

.ai 1 1

.am 24 23 1 1

.ao 2 2 1

.ar 1,255 1,223 20 12

.as 1 1 1

.at 961 935 21 5

.au 2,973 2,924 38 11

.az 33 31 1 1

.ba 115 110 4 1

.bd 3 3 1 1

.be 1,209 1,182 23 5

.bf 2 2 1 1

.bg 389 378 8 3

.bh 30 29 1 1

.bi 1 1 1

.biz 3 1 2 1

.bj 2 1 1

.bn 12 11 1 1

.bo 73 70 1 1

.br 5,415 5,314 68 34

.bs 10 10 1 1

.bw 5 5 1 1

.by 104 103 1 1

.bz 6 5 1 1

.ca 4,445 4,359 57 29

.cat 14 14 1 1

.cc 1 1 1 1

.cd 4 4 1

.cf 1 1

.cg 1 1 1

.ch 1,051 1,025 22 4

.ci 8 8 1 1

.ck 1 1 1

.cl 792 771 15 6

.cm 15 14 1 1

.cn 1 1 1

.co 989 971 12 6

.com 51,614 49,871 1,204 540

.cr 84 82 1 1

.cu 8 8 1 1

.cz 512 494 16 2

.de 9,360 9,219 100 41

.dj 2 2 1

.dk 535 524 10 2

.dm 5 4 1

.do 129 127 1 1

.dz 5 5 1 1

.ec 186 182 2 2

.edu 1 1 1

.ee 122 118 3 1

.eg 233 219 8 6

.es 4,330 4,221 82 27

.et 14 13 1 1

.eu 4 1 4 1

.fi 767 754 11 2

.fj 3 3 1

.fm 1 1 1

.fr 6,526 6,392 102 32

.ga 2 2 1

.gd 1 1

.ge 50 48 2 1

.gg 1 1 1

.gh 28 27 1 1

.gi 2 2

.gl 1 1 1

.gm 1 1 1 1

.gp 9 9 1 1

.gr 482 470 9 3

.gt 115 113 1 1

.gy 7 6 1 1

.hk 706 649 32 25

.hm 1 1

.hn 40 39 1 1

.hr 330 319 9 2

.ht 4 4 1

.hu 636 612 21 3

.id 738 694 38 6

.ie 490 485 4 2

.il 651 636 10 5

.im 1 1

.in 2,864 2,752 52 60

.io 1 1

.iq 20 18 1 1

.ir 1 1 1 1

.is 34 33 1 1

.it 5,505 5,390 87 28

.je 1 1

.jm 46 45 1 1

.jo 73 71 2 1

.jp 7,468 6,832 42 593

.ke 50 49 1 1

.kg 8 7 1 1

.kh 8 7 1 1

.ki 1 1

.kr 164 143 13 9

.kw 42 40 1 1

.kz 82 80 1 1

.la 2 2 1

.lb 40 39 1 1

.li 3 3 1

.lk 68 64 3 1

.ls 1 1 1

.lt 264 256 7 1

.lu 37 37 1 1

.lv 121 117 3 1

.ly 1 1 1 1

.ma 206 198 4 4

.md 54 53 1 1

.me 1 1 1 1

.mg 6 6 1 1

.mk 3 3 1

.ml 3 2 1 1

.mn 14 13 1 1

.ms 1 1 1

.mt 40 39 1 1

.mu 29 28 1 1

.mv 10 10 1 1

.mw 2 2

.mx 2,190 2,147 28 15

.my 568 546 18 4

.mz 13 13 1 1

.na 6 6 1 1

.ne 1 1

.net 28 4 22 2

.ng 68 66 1 1

.ni 24 24 1 1

.nl 1,906 1,862 37 7

.no 543 531 10 2

.np 21 20 1 1

.nr 1 1

.nu 1 1 1

.nz 495 485 8 2

.om 23 23 1 1

.org 254 5 247 2

.pa 19 19 1 1

.pe 664 644 10 11

.ph 655 640 12 4

.pk 324 309 8 7

.pl 3,997 3,906 76 15

.pn 1 1

.pr 109 107 2 1

.ps 29 27 1 1

.pt 778 760 14 4

.py 45 44 1 1

.qa 45 44 1 1

.ro 628 607 16 5

.rs 64 61 2 1

.ru 2,251 2,156 85 9

.rw 4 4 1

.sa 233 223 5 4

.sb 1 1

.sc 1 1

.se 1,242 1,219 20 3

.sg 498 488 7 3

.sh 1 1

.si 175 170 5 1

.sk 299 284 13 2

.sl 1 1 1

.sm 1 1 1 1

.sn 17 16 1 1

.so 1 1

.st 1 1 1

.su 1 1

.sv 68 66 1 1

.tg 1 1 1

.th 721 666 45 9

.tj 1 1

.tk 1 1 1

.tl 1 1 1

.tm 1 1

.tn 84 82 2 1

.to 1 1 1

.tr 1,448 1,404 30 14

.tt 53 52 1 1

.tv 1 1 1

.tw 408 394 8 6

.tz 17 16 1 1

.ua 919 901 11 6

.ug 15 14 1 1

.uk 8,731 8,589 107 34

.us 4 2 2 1

.uy 104 101 2 1

.uz 8 8 1 1

.vc 4 4 1

.ve 677 665 7 5

.vg 1 1

.vi 3 3 1 1

.vn 527 480 34 13

.vu 1 1

.ws 5 1 4 1

.za 362 352 7 3

.zm 4 4 1

.zw 10 10 1 1

undefined 27,377 23,094 825 3,457

Total 175,926 166,801 3,930 5,176

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges
Agents: as for genuine agent strings: too many crawlers indentify themselves as 'GoogleBot' to take this at face value. They are accepted as genuine Google crawler requests only when the ip address matches a known range (see above). Other records that mention GoogleBot are counted as GoogleBot? (question mark, as this may include partners, like DoCoMo). However when the agent string mentions Google Desktop or Google Earth this is always accepted
Service: the service name is based on the agent string (plus for GoogleBot check for ip address, see above), if this is inconclusive it is based on the referer string.
Here is detailed breakdown per service of indicators that pointed to Google (total ≥ 3)

Service Total Originating from
Google ip address Referer mentions
Google url Agent mentions
Google service

Desktop 852 - - Y
Earth 54 - - Y
FeedFetcher 331 Y - Y
GoogleBot 12,528 Y - Y
GoogleBot? 7,187 - - Y
GoogleBot? 64 - Y Y
Image search 53 - Y -
Mail 7 - Y -
Maps 178 - Y -
Other 389 - - Y
Other 32 - Y -
Other 95 - Y Y
Other 174 Y - -
Other 2,005 Y - Y
Other 3 Y Y Y
Toolbar 108 - - Y
Toolbar 39 - Y -
Toolbar 20 - Y Y
Translate 18 - Y -
Translate 815 Y - Y
Translate 213 Y Y Y
Web search 146,890 - Y -
Web search 13 Y Y -
Wireless 3,851 Y - Y

Top Level Domain 'undefined': requests with top level domain 'undefined' are nearly all requests from anonymous ip addresses (crawler and other services)
Note: averages below 1 are always rounded up to 1

Generated on Fri, Jun 3, 2011 13:18
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.