Wikimedia Visitor Log Analysis Report

Wikimedia Visitor Log Analysis Report - Google requests

Daily averages, based on sample period: 1 Jun 2010 - 29 Jun 2010

This analysis is based on a 1:1000 sampled server log (squids) ⇒ all counts x 1000.
See also: Requests by destination or by origin / Methods / Scripts / Skins / Crawlers / Op.Sys. / Browsers / Google

This report shows all requests to Wikimedia servers where a Google server of service was involved in any way, be it the GoogleBot crawler or FeedFetcher collector scripts that run on Google servers, or a user that follows a link from a Google Web or Google Desktop search results page, or from Google Maps or Google Earth etcetera.
Technically speaking three fields in the squid log records are checked for this: client ip address, referer header and user agent header. A request can originate from an ip address which has been registered by Google and/or it can carry a referer tag that tells us a user clicked a link on a Google results page and/or it can carry an agent string that mentions a Google application which can reasonably be assumed to be genuinely Google's. See bottom of page for further details.
In total Google was somehow involved in 42.8% of daily external page^* requests
Google referred to our sites, through its services including search, maps, and Google Earth, 107,436,000 page views per day, representing 38.4% of our external page requests.
Including all of its different search crawlers and services hosted on its servers, Google itself requested another 12,280,000 page pages per day, representing 4.4% of our external page requests.
* = mime type text/html only

In order of request volume

Requests originating from a Google ip address

Service Total Pages Images Other

GoogleBot 9,647 8,216 546 885

Wireless 3,114 3,082 29 2

Translate 677 676 1

Other 330 187 28 116

FeedFetcher 304 111 17 176

Web search 9 8 1 1

Toolbar 1 1

Desktop 1 1

KeywordTool 1 1

Total 14,081 12,280 620 1,179

Requests originating from elsewhere

Service Total Pages Images Other

Web search 108,965 104,951 2,328 1,686

GoogleBot? 2,375 2,157 6 211

Desktop 894 38 1 856

Toolbar 264 102 38 125

Earth 189 27 161 1

Other 165 69 14 81

Image search 83 34 48 1

Maps 76 36 39 1

Translate 37 4 26 7

Wireless 14 14

Mail 12 2 9 1

FeedFetcher 2 2 1

Total 113,076 107,436 2,669 2,969

Top level domains

.com 31,302 30,093 701 508

.de 8,181 7,977 143 61

.jp 6,005 5,693 103 209

.uk 5,733 5,577 85 71

.br 4,577 4,473 52 52

.fr 4,559 4,447 64 48

.it 4,172 4,073 57 42

.ca 3,665 3,557 59 50

.mx 3,423 3,344 40 39

.pl 3,273 3,208 39 25

.es 2,821 2,735 45 41

.au 2,568 2,502 41 25

.in 2,285 2,102 44 138

.ar 1,465 1,422 19 24

.nl 1,357 1,317 27 13

.co 1,288 1,262 12 14

.ru 1,211 1,106 94 11

.ph 1,097 1,052 29 16

.cl 1,076 1,046 18 12

.ve 1,062 1,037 11 14

.pe 1,014 963 12 38

.tr 874 844 20 9

.th 857 708 136 12

.be 827 807 11 9

.id 792 587 199 6

.ch 785 767 12 6

.at 781 760 14 7

.se 710 692 15 3

.hk 597 538 34 26

.fi 560 552 6 2

.my 535 452 76 7

.pt 531 519 7 5

.hu 497 480 13 4

.ro 487 468 11 7

.vn 443 368 70 4

.nz 435 415 7 12

.ua 434 425 4 5

.il 421 410 6 5

.dk 403 392 9 2

.sg 395 366 25 5

.ec 343 334 4 6

.gr 342 332 7 3

.bg 321 313 5 2

.no 306 299 6 2

.cz 298 289 7 2

.ie 280 273 3 4

.tw 274 265 4 5

.hr 239 232 5 2

.pk 202 185 4 12

.za 193 186 4 4

.sk 190 180 7 4

.gt 172 167 2 3

.lt 161 156 3 1

.eg 156 147 3 6

.cr 147 143 2 1

.do 145 140 2 3

.kr 140 95 28 17

.uy 137 133 2 2

.si 134 131 2 1

.sa 132 125 3 4

.ma 120 114 2 4

.sv 118 115 1 2

.ae 116 113 3 1

.bo 114 107 2 4

.pr 80 78 2 1

.ba 76 73 2 1

.ee 71 69 1 1

.lv 70 68 1 1

.py 70 68 1 1

.lk 58 55 2 2

.hn 51 49 1 1

.ke 47 45 1 1

.ng 44 42 1 2

.by 42 42 1 1

.rs 42 40 1 1

.ni 40 38 1 1

.tt 37 36 1 1

.kz 34 33 1 1

.jo 34 32 1 1

.qa 33 31 1 1

.jm 31 30 1 1

.lu 31 31 1 1

.kw 28 26 1 1

.md 28 28 1 1

.cn 25 24 1

.lb 25 24 1 1

.ge 23 22 1 1

.mt 22 21 1 1

.is 21 21 1 1

.bh 21 20 1 1

.np 19 17 1 1

.mu 19 19 1 1

.pa 18 17 1 1

.gh 16 16 1 1

.cu 16 16 1 1

.ps 14 13 1 1

.az 14 12 1 1

.ly 13 12 1 1

.om 13 12 1

.cm 12 11 1 1

.ug 11 11 1 1

.sn 10 10 1 1

.am 10 9 1 1

.et 10 9 1 1

.bn 10 10 1

.mv 9 9 1 1

.tz 9 8 1 1

.ci 8 8 1 1

.fj 8 7 1 1

.mz 8 7 1 1

.bs 8 7 1 1

.cat 8 8 1 1

.mn 7 7 1

.bd 6 6 1

.net 6 1 4 1

.org 6 1 4 1

.zw 6 6 1

.na 5 5 1 1

.rw 4 4 1 1

.bw 4 4

.mg 4 4

.kh 4 4 1

.bz 4 4 1 1

.gy 4 4 1

.dm 3 3 1

.kg 3 3 1 1

.dz 3 3 1 1

.uz 3 3 1

.gp 3 3 1

.biz 2 1 1

.af 2 2 1 1

.ga 2 1 1

.vg 2 2 1

.li 2 2

.cd 2 2 1 1

.ag 2 2 1

.vi 2 2

.ht 2 2 1 1

.zm 2 2 1 1

.vc 2 2 1

.gl 1 1

.bf 1 1

.ls 1 1 1

.gi 1 1

.la 1 1 1

.sb 1 1 1

.ai 1 1 1

.me 1 1 1

.us 1 1 1

.as 1 1 1 1

.sc 1 1

.ml 1 1

.bj 1 1 1 1

.gm 1 1 1

.to 1 1

.tl 1 1

.mk 1 1 1

.ws 1 1 1 1

.ad 1 1

.tg 1 1 1

.sm 1 1 1

.dj 1 1

.gg 1 1

.ne 1 1

.ki 1 1 1

.nu 1 1

.ck 1 1

.ao 1 1

.tm 1 1

.pn 1 1

.ir 1 1

.cg 1 1 1

.sl 1 1

.tv 1 1 1

.bi 1 1 1

.tk 1 1

.nr 1 1

.vu 1 1

.nf 1 1

.mw 1 1

.eu 1 1

.ac 1 1

.gov 1 1

.je 1 1

.sh 1 1

.im 1 1

.fm 1 1

undefined 17,414 14,206 771 2,437

Total 127,158 119,714 3,281 4,136

In alphabetical order

Requests originating from a Google ip address

Service Total Pages Images Other

Wireless 3,114 3,082 29 2

Web search 9 8 1 1

Translate 677 676 1

Toolbar 1 1

Other 330 187 28 116

KeywordTool 1 1

GoogleBot 9,647 8,216 546 885

FeedFetcher 304 111 17 176

Desktop 1 1

Total 14,081 12,280 620 1,179

Requests originating from elsewhere

Service Total Pages Images Other

Wireless 14 14

Web search 108,965 104,951 2,328 1,686

Translate 37 4 26 7

Toolbar 264 102 38 125

Other 165 69 14 81

Maps 76 36 39 1

Mail 12 2 9 1

Image search 83 34 48 1

GoogleBot? 2,375 2,157 6 211

FeedFetcher 2 2 1

Earth 189 27 161 1

Desktop 894 38 1 856

Total 113,076 107,436 2,669 2,969

Top level domains

.ac 1 1

.ad 1 1

.ae 116 113 3 1

.af 2 2 1 1

.ag 2 2 1

.ai 1 1 1

.am 10 9 1 1

.ao 1 1

.ar 1,465 1,422 19 24

.as 1 1 1 1

.at 781 760 14 7

.au 2,568 2,502 41 25

.az 14 12 1 1

.ba 76 73 2 1

.bd 6 6 1

.be 827 807 11 9

.bf 1 1

.bg 321 313 5 2

.bh 21 20 1 1

.bi 1 1 1

.biz 2 1 1

.bj 1 1 1 1

.bn 10 10 1

.bo 114 107 2 4

.br 4,577 4,473 52 52

.bs 8 7 1 1

.bw 4 4

.by 42 42 1 1

.bz 4 4 1 1

.ca 3,665 3,557 59 50

.cat 8 8 1 1

.cd 2 2 1 1

.cg 1 1 1

.ch 785 767 12 6

.ci 8 8 1 1

.ck 1 1

.cl 1,076 1,046 18 12

.cm 12 11 1 1

.cn 25 24 1

.co 1,288 1,262 12 14

.com 31,302 30,093 701 508

.cr 147 143 2 1

.cu 16 16 1 1

.cz 298 289 7 2

.de 8,181 7,977 143 61

.dj 1 1

.dk 403 392 9 2

.dm 3 3 1

.do 145 140 2 3

.dz 3 3 1 1

.ec 343 334 4 6

.ee 71 69 1 1

.eg 156 147 3 6

.es 2,821 2,735 45 41

.et 10 9 1 1

.eu 1 1

.fi 560 552 6 2

.fj 8 7 1 1

.fm 1 1

.fr 4,559 4,447 64 48

.ga 2 1 1

.ge 23 22 1 1

.gg 1 1

.gh 16 16 1 1

.gi 1 1

.gl 1 1

.gm 1 1 1

.gov 1 1

.gp 3 3 1

.gr 342 332 7 3

.gt 172 167 2 3

.gy 4 4 1

.hk 597 538 34 26

.hn 51 49 1 1

.hr 239 232 5 2

.ht 2 2 1 1

.hu 497 480 13 4

.id 792 587 199 6

.ie 280 273 3 4

.il 421 410 6 5

.im 1 1

.in 2,285 2,102 44 138

.ir 1 1

.is 21 21 1 1

.it 4,172 4,073 57 42

.je 1 1

.jm 31 30 1 1

.jo 34 32 1 1

.jp 6,005 5,693 103 209

.ke 47 45 1 1

.kg 3 3 1 1

.kh 4 4 1

.ki 1 1 1

.kr 140 95 28 17

.kw 28 26 1 1

.kz 34 33 1 1

.la 1 1 1

.lb 25 24 1 1

.li 2 2

.lk 58 55 2 2

.ls 1 1 1

.lt 161 156 3 1

.lu 31 31 1 1

.lv 70 68 1 1

.ly 13 12 1 1

.ma 120 114 2 4

.md 28 28 1 1

.me 1 1 1

.mg 4 4

.mk 1 1 1

.ml 1 1

.mn 7 7 1

.mt 22 21 1 1

.mu 19 19 1 1

.mv 9 9 1 1

.mw 1 1

.mx 3,423 3,344 40 39

.my 535 452 76 7

.mz 8 7 1 1

.na 5 5 1 1

.ne 1 1

.net 6 1 4 1

.nf 1 1

.ng 44 42 1 2

.ni 40 38 1 1

.nl 1,357 1,317 27 13

.no 306 299 6 2

.np 19 17 1 1

.nr 1 1

.nu 1 1

.nz 435 415 7 12

.om 13 12 1

.org 6 1 4 1

.pa 18 17 1 1

.pe 1,014 963 12 38

.ph 1,097 1,052 29 16

.pk 202 185 4 12

.pl 3,273 3,208 39 25

.pn 1 1

.pr 80 78 2 1

.ps 14 13 1 1

.pt 531 519 7 5

.py 70 68 1 1

.qa 33 31 1 1

.ro 487 468 11 7

.rs 42 40 1 1

.ru 1,211 1,106 94 11

.rw 4 4 1 1

.sa 132 125 3 4

.sb 1 1 1

.sc 1 1

.se 710 692 15 3

.sg 395 366 25 5

.sh 1 1

.si 134 131 2 1

.sk 190 180 7 4

.sl 1 1

.sm 1 1 1

.sn 10 10 1 1

.sv 118 115 1 2

.tg 1 1 1

.th 857 708 136 12

.tk 1 1

.tl 1 1

.tm 1 1

.to 1 1

.tr 874 844 20 9

.tt 37 36 1 1

.tv 1 1 1

.tw 274 265 4 5

.tz 9 8 1 1

.ua 434 425 4 5

.ug 11 11 1 1

.uk 5,733 5,577 85 71

.us 1 1 1

.uy 137 133 2 2

.uz 3 3 1

.vc 2 2 1

.ve 1,062 1,037 11 14

.vg 2 2 1

.vi 2 2

.vn 443 368 70 4

.vu 1 1

.ws 1 1 1 1

.za 193 186 4 4

.zm 2 2 1 1

.zw 6 6 1

undefined 17,414 14,206 771 2,437

Total 127,158 119,714 3,281 4,136

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges
Agents: as for genuine agent strings: too many crawlers indentify themselves as 'GoogleBot' to take this at face value. They are accepted as genuine Google crawler requests only when the ip address matches a known range (see above). Other records that mention GoogleBot are counted as GoogleBot? (question mark, as this may include partners, like DoCoMo). However when the agent string mentions Google Desktop or Google Earth this is always accepted
Service: the service name is based on the agent string (plus for GoogleBot check for ip address, see above), if this is inconclusive it is based on the referer string.
Here is detailed breakdown per service of indicators that pointed to Google (total ≥ 3)

Service Total Originating from
Google ip address Referer mentions
Google url Agent mentions
Google service

Desktop 893 - - Y
Earth 189 - - Y
FeedFetcher 304 Y - Y
GoogleBot 9,647 Y - Y
GoogleBot? 2,371 - - Y
GoogleBot? 4 - Y Y
Image search 83 - Y -
Mail 12 - Y -
Maps 76 - Y -
Other 152 - - Y
Other 7 - Y -
Other 6 - Y Y
Other 157 Y - -
Other 173 Y - Y
Toolbar 176 - - Y
Toolbar 57 - Y -
Toolbar 30 - Y Y
Translate 36 - Y -
Translate 494 Y - Y
Translate 183 Y Y Y
Web search 108,965 - Y -
Web search 9 Y Y -
Wireless 14 - - Y
Wireless 3,114 Y - Y

Top Level Domain 'undefined': requests with top level domain 'undefined' are nearly all requests from anonymous ip addresses (crawler and other services)
Note: averages below 1 are always rounded up to 1

Generated on Sat, Jul 3, 2010 19:20
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.