Wikimedia Visitor Log Analysis Report

Wikimedia Visitor Log Analysis Report - Google requests

Daily averages, based on sample period: 1 Apr 2010 - 30 Apr 2010

This analysis is based on a 1:1000 sampled server log (squids) ⇒ all counts x 1000.
See also: Requests by destination or by origin / Methods / Scripts / Skins / Crawlers / Op.Sys. / Browsers / Google

This report shows all requests to Wikimedia servers where a Google server of service was involved in any way, be it the GoogleBot crawler or FeedFetcher collector scripts that run on Google servers, or a user that follows a link from a Google Web or Google Desktop search results page, or from Google Maps or Google Earth etcetera.
Technically speaking three fields in the squid log records are checked for this: client ip address, referer header and user agent header. A request can originate from an ip address which has been registered by Google and/or it can carry a referer tag that tells us a user clicked a link on a Google results page and/or it can carry an agent string that mentions a Google application which can reasonably be assumed to be genuinely Google's. See bottom of page for further details.
In total Google was somehow involved in 43.0% of daily external page^* requests
Google referred to our sites, through its services including search, maps, and Google Earth, 115,663,000 page views per day, representing 38.0% of our external page requests.
Including all of its different search crawlers and services hosted on its servers, Google itself requested another 15,126,000 page pages per day, representing 5.0% of our external page requests.
* = mime type text/html only

In order of request volume

Requests originating from a Google ip address

Service Total Pages Images Other

GoogleBot 12,386 11,122 564 701

Wireless 2,912 2,883 27 2

Translate 725 725 1

Other 467 296 40 130

FeedFetcher 278 92 11 175

Web search 8 8 1 1

Image search 1 1

Toolbar 1 1

Desktop 1 1

Maps 1 1

Earth 1 1

KeywordTool 1 1

Total 16,776 15,126 642 1,008

Requests originating from elsewhere

Service Total Pages Images Other

Web search 112,296 110,490 1,381 425

GoogleBot? 3,325 3,146 9 170

Image search 3,115 1,626 1,462 26

Other 1,979 92 985 902

Desktop 1,248 60 1 1,189

Toolbar 929 144 36 749

Earth 333 48 284 1

Maps 96 49 46 1

Translate 43 4 32 7

Mail 14 2 11 1

Wireless 3 2 1 1

FeedFetcher 1 1 1

Total 123,382 115,663 4,246 3,471

Top level domains

.com 36,039 34,933 944 162

.de 8,068 7,883 156 29

.uk 5,924 5,804 100 19

.jp 5,412 5,298 99 15

.fr 4,871 4,766 86 19

.br 4,410 4,341 57 11

.ca 4,129 4,049 74 6

.it 3,992 3,918 63 11

.pl 3,900 3,831 56 13

.mx 3,431 3,377 47 7

.es 3,134 3,063 59 12

.au 2,307 2,263 39 6

.in 2,003 1,952 42 9

.co 1,918 1,857 21 40

.ar 1,412 1,385 22 5

.nl 1,345 1,310 30 5

.ru 1,248 1,132 113 3

.tr 1,135 1,097 34 4

.pe 1,040 1,022 13 5

.cl 1,035 1,015 16 5

.ve 940 926 10 4

.se 906 884 18 3

.id 906 670 230 6

.hk 825 793 28 3

.be 782 766 14 2

.at 766 747 16 3

.ch 756 740 14 2

.fi 717 707 9 2

.my 578 491 85 2

.pt 553 541 10 2

.hu 537 517 17 3

.ph 537 512 23 3

.vn 533 443 89 2

.ro 491 474 15 2

.th 485 381 99 5

.il 448 437 8 2

.ua 448 439 7 2

.sg 426 398 26 2

.dk 416 405 10 1

.nz 408 394 10 4

.no 399 388 9 2

.ec 350 345 3 1

.gr 342 332 9 2

.ie 340 334 4 1

.cz 323 311 10 1

.tw 311 304 6 1

.bg 307 300 7 1

.hr 260 253 6 1

.gt 225 221 3 1

.za 224 218 6 1

.sk 215 204 8 3

.lt 199 194 4 1

.eg 193 187 6 1

.do 192 189 2 1

.pk 189 183 5 1

.cr 160 157 2 1

.ma 146 142 3 1

.si 140 136 3 1

.sv 139 136 2 1

.uy 139 136 2 1

.kr 139 105 34 1

.bo 136 134 2 1

.ae 132 128 4 1

.pr 132 130 2 1

.sa 128 123 5 1

.lv 98 96 2 1

.ee 97 96 1 1

.ba 84 80 3 1

.py 66 65 1 1

.hn 65 63 1 1

.jo 52 51 2 1

.lk 46 44 2 1

.by 44 43 1 1

.rs 43 41 1 1

.kz 42 42 1 1

.ni 41 40 1 1

.ke 39 38 1 1

.ng 39 38 1 1

.tt 39 39 1 1

.jm 38 38 1 1

.md 36 35 1 1

.is 32 31 1 1

.kw 32 32 1

.lu 30 29 1 1

.qa 30 29 1 1

.cn 28 27 1 1

.lb 26 26 1

.mt 25 25 1 1

.bh 23 22 1 1

.ge 22 21 1 1

.om 22 22 1 1

.mu 21 21 1 1

.ps 19 18 1 1

.pa 18 18 1

.gh 18 18 1 1

.az 17 16 1 1

.ug 13 12 1 1

.am 13 12 1 1

.np 13 13 1 1

.ly 12 11 1

.cu 12 12 1

.bn 12 11 1 1

.ci 11 11 1 1

.sn 11 11 1 1

.cm 11 11 1 1

.mn 11 11 1 1

.mv 9 9 1 1

.bs 9 9 1

.et 9 9 1 1

.tz 8 7 1

.dz 8 8 1 1

.mz 8 8 1 1

.cat 8 8 1 1

.fj 7 7 1 1

.bd 6 6 1

.net 6 1 5 1

.bw 5 5 1

.mg 5 5 1 1

.bz 5 5 1 1

.org 5 1 4 1

.zw 5 5 1

.dm 4 4

.na 4 4 1

.kh 4 3 1

.uz 4 4 1 1

.gp 4 3 1

.rw 3 3

.kg 3 3

.ag 3 3 1

.vi 3 3

.gy 3 3 1

.zm 3 3

.vc 3 3

.bf 2 1 1

.af 2 2 1

.ls 2 2 1

.gi 2 2 1

.li 2 2

.cd 2 2 1

.ml 2 2 1 1

.ht 2 2 1

.biz 1 1 1 1

.gl 1 1

.la 1 1 1

.ga 1 1 1

.ai 1 1

.me 1 1 1

.vg 1 1 1

.as 1 1

.sc 1 1

.bj 1 1 1

.gm 1 1 1

.to 1 1 1

.mk 1 1 1

.ad 1 1

.tg 1 1

.sm 1 1

.dj 1 1

.gg 1 1

.ne 1 1

.ki 1 1

.nu 1 1 1

.st 1 1

.ck 1 1 1

.ao 1 1

.tm 1 1

.ms 1 1

.pn 1 1

.sb 1 1 1

.ir 1 1 1

.cg 1 1

.sl 1 1

.us 1 1

.tv 1 1 1

.bi 1 1

.tk 1 1 1

.su 1 1

.nr 1 1

.vu 1 1

.edu 1 1 1

.mw 1 1

.eu 1 1

.ac 1 1

.je 1 1

.cc 1 1

.sh 1 1

.tl 1 1

.cf 1 1

.im 1 1 1

.fm 1 1

.ws 1 1 1

undefined 23,954 18,056 1,894 4,005

Total 140,153 130,789 4,879 4,473

In alphabetical order

Requests originating from a Google ip address

Service Total Pages Images Other

Wireless 2,912 2,883 27 2

Web search 8 8 1 1

Translate 725 725 1

Toolbar 1 1

Other 467 296 40 130

Maps 1 1

KeywordTool 1 1

Image search 1 1

GoogleBot 12,386 11,122 564 701

FeedFetcher 278 92 11 175

Earth 1 1

Desktop 1 1

Total 16,776 15,126 642 1,008

Requests originating from elsewhere

Service Total Pages Images Other

Wireless 3 2 1 1

Web search 112,296 110,490 1,381 425

Translate 43 4 32 7

Toolbar 929 144 36 749

Other 1,979 92 985 902

Maps 96 49 46 1

Mail 14 2 11 1

Image search 3,115 1,626 1,462 26

GoogleBot? 3,325 3,146 9 170

FeedFetcher 1 1 1

Earth 333 48 284 1

Desktop 1,248 60 1 1,189

Total 123,382 115,663 4,246 3,471

Top level domains

.ac 1 1

.ad 1 1

.ae 132 128 4 1

.af 2 2 1

.ag 3 3 1

.ai 1 1

.am 13 12 1 1

.ao 1 1

.ar 1,412 1,385 22 5

.as 1 1

.at 766 747 16 3

.au 2,307 2,263 39 6

.az 17 16 1 1

.ba 84 80 3 1

.bd 6 6 1

.be 782 766 14 2

.bf 2 1 1

.bg 307 300 7 1

.bh 23 22 1 1

.bi 1 1

.biz 1 1 1 1

.bj 1 1 1

.bn 12 11 1 1

.bo 136 134 2 1

.br 4,410 4,341 57 11

.bs 9 9 1

.bw 5 5 1

.by 44 43 1 1

.bz 5 5 1 1

.ca 4,129 4,049 74 6

.cat 8 8 1 1

.cc 1 1

.cd 2 2 1

.cf 1 1

.cg 1 1

.ch 756 740 14 2

.ci 11 11 1 1

.ck 1 1 1

.cl 1,035 1,015 16 5

.cm 11 11 1 1

.cn 28 27 1 1

.co 1,918 1,857 21 40

.com 36,039 34,933 944 162

.cr 160 157 2 1

.cu 12 12 1

.cz 323 311 10 1

.de 8,068 7,883 156 29

.dj 1 1

.dk 416 405 10 1

.dm 4 4

.do 192 189 2 1

.dz 8 8 1 1

.ec 350 345 3 1

.edu 1 1 1

.ee 97 96 1 1

.eg 193 187 6 1

.es 3,134 3,063 59 12

.et 9 9 1 1

.eu 1 1

.fi 717 707 9 2

.fj 7 7 1 1

.fm 1 1

.fr 4,871 4,766 86 19

.ga 1 1 1

.ge 22 21 1 1

.gg 1 1

.gh 18 18 1 1

.gi 2 2 1

.gl 1 1

.gm 1 1 1

.gp 4 3 1

.gr 342 332 9 2

.gt 225 221 3 1

.gy 3 3 1

.hk 825 793 28 3

.hn 65 63 1 1

.hr 260 253 6 1

.ht 2 2 1

.hu 537 517 17 3

.id 906 670 230 6

.ie 340 334 4 1

.il 448 437 8 2

.im 1 1 1

.in 2,003 1,952 42 9

.ir 1 1 1

.is 32 31 1 1

.it 3,992 3,918 63 11

.je 1 1

.jm 38 38 1 1

.jo 52 51 2 1

.jp 5,412 5,298 99 15

.ke 39 38 1 1

.kg 3 3

.kh 4 3 1

.ki 1 1

.kr 139 105 34 1

.kw 32 32 1

.kz 42 42 1 1

.la 1 1 1

.lb 26 26 1

.li 2 2

.lk 46 44 2 1

.ls 2 2 1

.lt 199 194 4 1

.lu 30 29 1 1

.lv 98 96 2 1

.ly 12 11 1

.ma 146 142 3 1

.md 36 35 1 1

.me 1 1 1

.mg 5 5 1 1

.mk 1 1 1

.ml 2 2 1 1

.mn 11 11 1 1

.ms 1 1

.mt 25 25 1 1

.mu 21 21 1 1

.mv 9 9 1 1

.mw 1 1

.mx 3,431 3,377 47 7

.my 578 491 85 2

.mz 8 8 1 1

.na 4 4 1

.ne 1 1

.net 6 1 5 1

.ng 39 38 1 1

.ni 41 40 1 1

.nl 1,345 1,310 30 5

.no 399 388 9 2

.np 13 13 1 1

.nr 1 1

.nu 1 1 1

.nz 408 394 10 4

.om 22 22 1 1

.org 5 1 4 1

.pa 18 18 1

.pe 1,040 1,022 13 5

.ph 537 512 23 3

.pk 189 183 5 1

.pl 3,900 3,831 56 13

.pn 1 1

.pr 132 130 2 1

.ps 19 18 1 1

.pt 553 541 10 2

.py 66 65 1 1

.qa 30 29 1 1

.ro 491 474 15 2

.rs 43 41 1 1

.ru 1,248 1,132 113 3

.rw 3 3

.sa 128 123 5 1

.sb 1 1 1

.sc 1 1

.se 906 884 18 3

.sg 426 398 26 2

.sh 1 1

.si 140 136 3 1

.sk 215 204 8 3

.sl 1 1

.sm 1 1

.sn 11 11 1 1

.st 1 1

.su 1 1

.sv 139 136 2 1

.tg 1 1

.th 485 381 99 5

.tk 1 1 1

.tl 1 1

.tm 1 1

.to 1 1 1

.tr 1,135 1,097 34 4

.tt 39 39 1 1

.tv 1 1 1

.tw 311 304 6 1

.tz 8 7 1

.ua 448 439 7 2

.ug 13 12 1 1

.uk 5,924 5,804 100 19

.us 1 1

.uy 139 136 2 1

.uz 4 4 1 1

.vc 3 3

.ve 940 926 10 4

.vg 1 1 1

.vi 3 3

.vn 533 443 89 2

.vu 1 1

.ws 1 1 1

.za 224 218 6 1

.zm 3 3

.zw 5 5 1

undefined 23,954 18,056 1,894 4,005

Total 140,153 130,789 4,879 4,473

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges
Agents: as for genuine agent strings: too many crawlers indentify themselves as 'GoogleBot' to take this at face value. They are accepted as genuine Google crawler requests only when the ip address matches a known range (see above). Other records that mention GoogleBot are counted as GoogleBot? (question mark, as this may include partners, like DoCoMo). However when the agent string mentions Google Desktop or Google Earth this is always accepted
Service: the service name is based on the agent string (plus for GoogleBot check for ip address, see above), if this is inconclusive it is based on the referer string.
Here is detailed breakdown per service of indicators that pointed to Google (total ≥ 3)

Service Total Originating from
Google ip address Referer mentions
Google url Agent mentions
Google service

Desktop 1,248 - - Y
Earth 333 - - Y
FeedFetcher 278 Y - Y
GoogleBot 12,386 Y - Y
GoogleBot? 3,319 - - Y
GoogleBot? 5 - Y Y
Image search 3,115 - Y -
Mail 14 - Y -
Maps 96 - Y -
Other 114 - - Y
Other 1,864 - Y -
Other 199 Y - -
Other 268 Y - Y
Toolbar 819 - - Y
Toolbar 73 - Y -
Toolbar 37 - Y Y
Translate 43 - Y -
Translate 526 Y - Y
Translate 199 Y Y Y
Web search 112,296 - Y -
Web search 8 Y Y -
Wireless 3 - - Y
Wireless 2,912 Y - Y

Top Level Domain 'undefined': requests with top level domain 'undefined' are nearly all requests from anonymous ip addresses (crawler and other services)
Note: averages below 1 are always rounded up to 1

Generated on Sat, Jun 26, 2010 9:26
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.