Wikimedia Traffic Analysis Report

Wikimedia Traffic Analysis Report - Google requests

Daily averages, based on sample period: 1 Jun 2011 - 30 Jun 2011

This analysis is based on a 1:1000 sampled server log (squids) ⇒ all counts x 1000.
See also: Requests by destination or by origin / Methods / Scripts / Skins / Crawlers / Op.Sys. / Browsers / Google

This report shows all requests to Wikimedia servers where a Google server of service was involved in any way, be it the GoogleBot crawler or FeedFetcher collector scripts that run on Google servers, or a user that follows a link from a Google Web or Google Desktop search results page, or from Google Maps or Google Earth etcetera.
Technically speaking three fields in the squid log records are checked for this: client ip address, referer header and user agent header. A request can originate from an ip address which has been registered by Google and/or it can carry a referer tag that tells us a user clicked a link on a Google results page and/or it can carry an agent string that mentions a Google application which can reasonably be assumed to be genuinely Google's. See bottom of page for further details.
In total Google was somehow involved in 40.9% of daily external page^* requests
Google referred to our sites, through its services including search, maps, and Google Earth, 139,448,000 page views per day, representing 35.5% of our external page requests.
Including all of its different search crawlers and services hosted on its servers, Google itself requested another 21,378,000 page pages per day, representing 5.4% of our external page requests.
* = mime type text/html only

In order of request volume

Requests originating from a Google ip address

Service Total Pages Images Other

GoogleBot 17,483 15,611 928 944

Wireless 3,713 3,645 64 4

Other 2,416 1,015 398 1,003

Translate 1,006 1,005 1 1

FeedFetcher 357 89 28 240

Web search 15 13 1

Image search 1 1

Toolbar 1 1

Desktop 1 1

Maps 1 1

Total 24,990 21,378 1,419 2,191

Requests originating from elsewhere

Service Total Pages Images Other

Web search 136,104 132,251 2,549 1,304

GoogleBot? 7,560 6,900 38 621

Desktop 767 21 746

Other 478 131 101 246

Maps 162 33 128 1

Toolbar 138 51 15 71

Image search 55 30 25 1

Earth 52 22 30 1

Translate 15 4 9 2

Mail 7 2 5 1

Wireless 2 1 1 1

FeedFetcher 2 2 1 1

Total 145,342 139,448 2,901 2,991

Top level domains

.com 46,853 45,276 1,237 340

.de 8,644 8,523 91 31

.uk 8,080 7,896 150 34

.jp 7,398 6,775 43 579

.fr 5,722 5,584 115 23

.it 5,416 5,284 109 23

.br 5,155 5,044 90 21

.ca 4,139 4,048 72 20

.es 3,694 3,583 91 20

.pl 3,517 3,433 72 12

.in 3,253 3,102 89 62

.au 2,878 2,815 53 10

.ru 2,057 1,987 62 8

.mx 1,980 1,936 35 9

.nl 1,884 1,830 48 6

.ph 1,446 1,412 26 9

.ar 1,168 1,136 25 7

.be 1,127 1,097 27 4

.tr 1,124 1,087 29 8

.ch 1,030 1,001 25 4

.th 1,016 961 44 12

.se 945 925 17 2

.at 915 889 22 4

.ua 825 809 12 4

.cl 774 753 17 4

.pt 703 685 16 2

.pe 684 660 16 8

.hk 672 633 16 24

.id 666 627 35 4

.co 640 627 11 2

.fi 626 613 11 1

.ve 619 605 10 4

.il 611 594 12 5

.my 603 581 18 4

.hu 573 550 20 3

.ro 563 543 17 3

.sg 496 486 6 3

.vn 496 463 21 11

.nz 490 481 9 1

.dk 483 473 10 1

.gr 450 437 11 2

.cz 437 419 17 1

.ie 431 425 4 2

.no 423 411 11 1

.tw 402 389 8 5

.bg 377 367 9 2

.pk 303 287 10 6

.za 292 282 8 2

.hr 278 269 8 2

.sk 252 242 9 1

.eg 216 202 9 5

.sa 201 193 6 2

.ma 178 171 4 3

.lt 177 171 6 1

.org 176 4 171 1

.kr 165 145 11 9

.ec 162 158 3 1

.ae 158 155 3 1

.si 157 152 5 1

.gt 106 102 2 2

.ba 102 97 4 1

.by 101 99 1 1

.do 101 99 1 1

.uy 94 92 2 1

.ee 87 85 2 1

.lv 85 82 3 1

.cr 84 82 2 1

.pr 84 82 2 1

.ng 74 71 1 2

.lk 73 68 4 1

.bo 69 66 2 1

.sv 61 60 1 1

.ke 59 58 1 1

.tn 57 55 2 1

.rs 57 55 1 1

.jo 50 48 2 1

.tt 49 48 1 1

.md 47 46 1 1

.py 44 43 1 1

.ge 43 40 2 1

.kz 41 41 1 1

.lu 40 39 1 1

.qa 40 39 1 1

.jm 39 38 1 1

.hn 36 35 1 1

.kw 35 34 1 1

.mt 33 33 1 1

.az 30 28 1 1

.lb 29 27 1 1

.is 28 28 1 1

.bh 26 25 1

.gh 26 25 1 1

.np 25 23 1 1

.mu 24 24 1 1

.net 24 3 19 1

.ni 23 23 1 1

.am 21 20 1 1

.ps 20 19 1 1

.ws 20 1 20

.iq 19 17 2 1

.pa 19 18 1 1

.om 17 17 1 1

.ug 15 14 1 1

.tz 15 15 1 1

.cm 13 12 1 1

.ci 12 12 1 1

.sn 12 11 1 1

.bn 12 11 1

.mv 11 10 1 1

.mz 11 11 1 1

.et 11 11 1 1

.mn 10 9 1 1

.cat 10 10 1 1

.zw 9 9 1 1

.kh 8 7 1 1

.bs 8 8 1 1

.gp 8 7 1 1

.mg 7 6 1 1

.cu 7 7 1

.dm 6 6 1 1

.na 6 6 1

.kg 6 6 1 1

.uz 6 6 1 1

.rw 5 4 1 1

.dz 5 4 1 1

.bw 5 5 1 1

.gy 5 5 1 1

.cd 4 3 1 1

.bz 4 4 1

.ht 4 4

.zm 4 4 1

.bd 3 3 1 1

.af 3 3 1 1

.ag 3 3 1

.ml 3 3 1 1

.vc 3 3 1 1

.biz 2 1 1 1

.bf 2 2 1

.ao 2 2 1

.ga 2 1 1 1

.gi 2 2

.la 2 2 1 1

.fj 2 2 1

.li 2 2 1

.vi 2 2

.mw 2 2

.eu 2 1 2 1

.mk 2 2 1

.ne 1 1

.ls 1 1 1

.tm 1 1 1

.ly 1 1 1

.ai 1 1 1

.me 1 1 1 1

.vg 1 1

.ir 1 1 1

.sl 1 1

.us 1 1 1 1

.bi 1 1

.as 1 1

.sc 1 1 1

.bj 1 1 1

.gm 1 1

.to 1 1 1 1

.cc 1 1 1

.tl 1 1 1

.fm 1 1 1

.ad 1 1 1

.tg 1 1 1

.sm 1 1 1

.dj 1 1 1

.gg 1 1 1

.ki 1 1

.gl 1 1

.nu 1 1 1

.st 1 1 1

.hm 1 1

.ck 1 1

.ms 1 1

.pn 1 1

.sb 1 1 1 1

.gs 1 1

.cg 1 1

.tv 1 1

.tk 1 1 1

.su 1 1

.cn 1 1 1 1

.cx 1 1

.nr 1 1

.vu 1 1

.edu 1 1 1

.ac 1 1

.je 1 1

.sh 1 1

.so 1 1 1

.tj 1 1

.cf 1 1

.im 1 1 1

.mil 1 1

undefined 32,687 27,821 1,076 3,790

Total 170,330 160,823 4,320 5,179

In alphabetical order

Requests originating from a Google ip address

Service Total Pages Images Other

Wireless 3,713 3,645 64 4

Web search 15 13 1

Translate 1,006 1,005 1 1

Toolbar 1 1

Other 2,416 1,015 398 1,003

Maps 1 1

Image search 1 1

GoogleBot 17,483 15,611 928 944

FeedFetcher 357 89 28 240

Desktop 1 1

Total 24,990 21,378 1,419 2,191

Requests originating from elsewhere

Service Total Pages Images Other

Wireless 2 1 1 1

Web search 136,104 132,251 2,549 1,304

Translate 15 4 9 2

Toolbar 138 51 15 71

Other 478 131 101 246

Maps 162 33 128 1

Mail 7 2 5 1

Image search 55 30 25 1

GoogleBot? 7,560 6,900 38 621

FeedFetcher 2 2 1 1

Earth 52 22 30 1

Desktop 767 21 746

Total 145,342 139,448 2,901 2,991

Top level domains

.ac 1 1

.ad 1 1 1

.ae 158 155 3 1

.af 3 3 1 1

.ag 3 3 1

.ai 1 1 1

.am 21 20 1 1

.ao 2 2 1

.ar 1,168 1,136 25 7

.as 1 1

.at 915 889 22 4

.au 2,878 2,815 53 10

.az 30 28 1 1

.ba 102 97 4 1

.bd 3 3 1 1

.be 1,127 1,097 27 4

.bf 2 2 1

.bg 377 367 9 2

.bh 26 25 1

.bi 1 1

.biz 2 1 1 1

.bj 1 1 1

.bn 12 11 1

.bo 69 66 2 1

.br 5,155 5,044 90 21

.bs 8 8 1 1

.bw 5 5 1 1

.by 101 99 1 1

.bz 4 4 1

.ca 4,139 4,048 72 20

.cat 10 10 1 1

.cc 1 1 1

.cd 4 3 1 1

.cf 1 1

.cg 1 1

.ch 1,030 1,001 25 4

.ci 12 12 1 1

.ck 1 1

.cl 774 753 17 4

.cm 13 12 1 1

.cn 1 1 1 1

.co 640 627 11 2

.com 46,853 45,276 1,237 340

.cr 84 82 2 1

.cu 7 7 1

.cx 1 1

.cz 437 419 17 1

.de 8,644 8,523 91 31

.dj 1 1 1

.dk 483 473 10 1

.dm 6 6 1 1

.do 101 99 1 1

.dz 5 4 1 1

.ec 162 158 3 1

.edu 1 1 1

.ee 87 85 2 1

.eg 216 202 9 5

.es 3,694 3,583 91 20

.et 11 11 1 1

.eu 2 1 2 1

.fi 626 613 11 1

.fj 2 2 1

.fm 1 1 1

.fr 5,722 5,584 115 23

.ga 2 1 1 1

.ge 43 40 2 1

.gg 1 1 1

.gh 26 25 1 1

.gi 2 2

.gl 1 1

.gm 1 1

.gp 8 7 1 1

.gr 450 437 11 2

.gs 1 1

.gt 106 102 2 2

.gy 5 5 1 1

.hk 672 633 16 24

.hm 1 1

.hn 36 35 1 1

.hr 278 269 8 2

.ht 4 4

.hu 573 550 20 3

.id 666 627 35 4

.ie 431 425 4 2

.il 611 594 12 5

.im 1 1 1

.in 3,253 3,102 89 62

.iq 19 17 2 1

.ir 1 1 1

.is 28 28 1 1

.it 5,416 5,284 109 23

.je 1 1

.jm 39 38 1 1

.jo 50 48 2 1

.jp 7,398 6,775 43 579

.ke 59 58 1 1

.kg 6 6 1 1

.kh 8 7 1 1

.ki 1 1

.kr 165 145 11 9

.kw 35 34 1 1

.kz 41 41 1 1

.la 2 2 1 1

.lb 29 27 1 1

.li 2 2 1

.lk 73 68 4 1

.ls 1 1 1

.lt 177 171 6 1

.lu 40 39 1 1

.lv 85 82 3 1

.ly 1 1 1

.ma 178 171 4 3

.md 47 46 1 1

.me 1 1 1 1

.mg 7 6 1 1

.mil 1 1

.mk 2 2 1

.ml 3 3 1 1

.mn 10 9 1 1

.ms 1 1

.mt 33 33 1 1

.mu 24 24 1 1

.mv 11 10 1 1

.mw 2 2

.mx 1,980 1,936 35 9

.my 603 581 18 4

.mz 11 11 1 1

.na 6 6 1

.ne 1 1

.net 24 3 19 1

.ng 74 71 1 2

.ni 23 23 1 1

.nl 1,884 1,830 48 6

.no 423 411 11 1

.np 25 23 1 1

.nr 1 1

.nu 1 1 1

.nz 490 481 9 1

.om 17 17 1 1

.org 176 4 171 1

.pa 19 18 1 1

.pe 684 660 16 8

.ph 1,446 1,412 26 9

.pk 303 287 10 6

.pl 3,517 3,433 72 12

.pn 1 1

.pr 84 82 2 1

.ps 20 19 1 1

.pt 703 685 16 2

.py 44 43 1 1

.qa 40 39 1 1

.ro 563 543 17 3

.rs 57 55 1 1

.ru 2,057 1,987 62 8

.rw 5 4 1 1

.sa 201 193 6 2

.sb 1 1 1 1

.sc 1 1 1

.se 945 925 17 2

.sg 496 486 6 3

.sh 1 1

.si 157 152 5 1

.sk 252 242 9 1

.sl 1 1

.sm 1 1 1

.sn 12 11 1 1

.so 1 1 1

.st 1 1 1

.su 1 1

.sv 61 60 1 1

.tg 1 1 1

.th 1,016 961 44 12

.tj 1 1

.tk 1 1 1

.tl 1 1 1

.tm 1 1 1

.tn 57 55 2 1

.to 1 1 1 1

.tr 1,124 1,087 29 8

.tt 49 48 1 1

.tv 1 1

.tw 402 389 8 5

.tz 15 15 1 1

.ua 825 809 12 4

.ug 15 14 1 1

.uk 8,080 7,896 150 34

.us 1 1 1 1

.uy 94 92 2 1

.uz 6 6 1 1

.vc 3 3 1 1

.ve 619 605 10 4

.vg 1 1

.vi 2 2

.vn 496 463 21 11

.vu 1 1

.ws 20 1 20

.za 292 282 8 2

.zm 4 4 1

.zw 9 9 1 1

undefined 32,687 27,821 1,076 3,790

Total 170,330 160,823 4,320 5,179

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges
Agents: as for genuine agent strings: too many crawlers indentify themselves as 'GoogleBot' to take this at face value. They are accepted as genuine Google crawler requests only when the ip address matches a known range (see above). Other records that mention GoogleBot are counted as GoogleBot? (question mark, as this may include partners, like DoCoMo). However when the agent string mentions Google Desktop or Google Earth this is always accepted
Service: the service name is based on the agent string (plus for GoogleBot check for ip address, see above), if this is inconclusive it is based on the referer string.
Here is detailed breakdown per service of indicators that pointed to Google (total ≥ 3)

Service Total Originating from
Google ip address Referer mentions
Google url Agent mentions
Google service

Desktop 767 - - Y
Earth 52 - - Y
FeedFetcher 357 Y - Y
GoogleBot 17,481 Y - Y
GoogleBot? 7,490 - - Y
GoogleBot? 69 - Y Y
Image search 55 - Y -
Mail 7 - Y -
Maps 162 - Y -
Other 354 - - Y
Other 36 - Y -
Other 88 - Y Y
Other 234 Y - -
Other 2,180 Y - Y
Toolbar 90 - - Y
Toolbar 32 - Y -
Toolbar 16 - Y Y
Translate 13 - Y -
Translate 806 Y - Y
Translate 199 Y Y Y
Web search 136,104 - Y -
Web search 15 Y Y -
Wireless 3,713 Y - Y

Top Level Domain 'undefined': requests with top level domain 'undefined' are nearly all requests from anonymous ip addresses (crawler and other services)
Note: averages below 1 are always rounded up to 1

Generated on Fri, Jul 15, 2011 22:17
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.