udp2log message loss, and traffic volume, based on 1:1000 sampled squid logs

Server roles mostly taken from squid.php

api/eqiad:cp1001, cp1002, cp1003, cp1004, cp1005,
api/pmpta:sq33, sq34, sq36,
mobile/eqiad:cp1041, cp1042, cp1046, cp1047, cp1059, cp1060,
mobile/esams:cp3011, cp3012, cp3013, cp3014,
mobile/pmpta:cp1041, cp1042, cp1043, cp1044,
ssl/eqiad:ssl1001, ssl1002, ssl1003, ssl1004, ssl1005, ssl1006, ssl1007, ssl1008, ssl1009,
ssl/esams:ssl3001, ssl3002, ssl3003, ssl3004,
ssl/pmpta:ssl1, ssl2, ssl3, ssl4,
text/eqiad:cp1006, cp1007, cp1008, cp1009, cp1010, cp1011, cp1012, cp1013, cp1014, cp1015, cp1016, cp1017, cp1018, cp1019, cp1020,
text/esams:amssq31, amssq32, amssq33, amssq34, amssq35, amssq36, amssq37, amssq38, amssq39, amssq40, amssq41, amssq42, amssq43, amssq44, amssq45, amssq46, knsq23, knsq24, knsq26, knsq27, knsq28, knsq29,
text/knams:knsq23, knsq24, knsq25, knsq26, knsq27, knsq28, knsq29,
text/pmpta:sq37, sq59, sq60, sq61, sq62, sq63, sq64, sq65, sq66, sq71, sq72, sq73, sq74, sq75, sq76, sq77, sq78,
upload/eqiad:cp1021, cp1022, cp1023, cp1024, cp1025, cp1026, cp1027, cp1028, cp1029, cp1030, cp1031, cp1032, cp1033, cp1034, cp1035, cp1036, cp1048, cp1049, cp1050, cp1051, cp1061, cp1062, cp1063, cp1064, dysprosium,
upload/esams:amssq47, amssq48, amssq49, amssq50, amssq51, amssq52, amssq53, amssq54, amssq55, amssq56, amssq57, amssq58, amssq59, amssq60, amssq61, amssq62, cp3003, cp3004, cp3005, cp3006, cp3007, cp3008, cp3009, cp3010, knsq16, knsq17, knsq18, knsq19, knsq20, knsq21, knsq22,
upload/knams:knsq16, knsq17, knsq18, knsq19, knsq2, knsq20, knsq21, knsq22,
upload/pmpta:sq41, sq42, sq43, sq44, sq45, sq48, sq49, sq50, sq51, sq52, sq53, sq54, sq55, sq56, sq57, sq58, sq79, sq80, sq81, sq82, sq83, sq84, sq85, sq86,
varnish/eqiad:cp1037, cp1038, cp1039, cp1040, cp1052, cp1053, cp1054, cp1055, cp1065, cp1066, cp1067, cp1068,
role?/loc.?:amssq47, cp1043, cp1044, cp4001, cp4002, cp4003, cp4004, cp4005, cp4005.ulsfo, cp4006, cp4006.ulsfo, cp4007, cp4007.ulsfo, cp4008, cp4008.ulsfo, cp4009, cp4009.ulsfo, cp4010, cp4010.ulsfo, cp4011, cp4011.ulsfo, cp4012, cp4012.ulsfo, cp4013, cp4013.ulsfo, cp4014, cp4014.ulsfo, cp4015, cp4015.ulsfo, cp4016, cp4016.ulsfo, cp4017, cp4017.ulsfo, cp4018, cp4018.ulsfo, cp4019, cp4019.ulsfo, cp4020, cp4020.ulsfo, deployment-cache-bits03, deployment-cache-mobile01, deployment-cache-text1, i-000000dc,
role?/pmpta:holmium, marmontel,
~RIP1/pmpta:sq31, sq32, sq35, sq38, sq39, sq40, sq46, sq47,
~RIP2/knams:knsq1, knsq10, knsq11, knsq12, knsq13, knsq14, knsq15, knsq3, knsq30, knsq4, knsq5, knsq6, knsq7, knsq8, knsq9,
~RIP3/pmpta:mobile1, mobile2, mobile3, mobile4, mobile5,
~RIP5/loc.?:gurvin, maerlant, yvon,

Legend

Packet loss less than 0.2% (avg gap < 1020)

Packet loss between 0.2% and 0.5% (1020 <= avg gap < 1052)

Packet loss larger than 0.5% (avg gap >= 1052)

Insignificant message volume: less than 10 (sampled) msgs per hour

Sequence numbers totally broken

Msg loss is weighed average over all servers, excluding those where sequence numbers are broken (weighed = taking into account message volume)
Other causes of data loss than UDP msg loss are not covered by this report (e.g. total outage for several hours)
 

UDP message loss: average gap between sequence numbers per squid (should be close to 1000 in 1:1000 sampled log) For calculation of message loss percentage from average gap between sequence numbers see bottom of page
		api	api	mobile	mobile	mobile	ssl	ssl	ssl	text	text	text	text	text	upload	upload	upload	upload	upload	upload	upload	varnish	role?	RIP1	RIP2	RIP3	RIP5
		eqiad	pmpta	eqiad	esams	pmpta	eqiad	esams	pmpta	eqiad	esams	esams	knams	pmpta	eqiad	eqiad	esams	esams	esams	knams	pmpta	eqiad	loc	pmpta	knams	pmpta	loc
month	msg loss	cp1001-1005	sq33-36	cp1041-1060	cp3011-3014	cp1041-1044	ssl1001-1009	ssl3001-3004	ssl1-4	cp1006-1020	amssq31-46	knsq23-29	knsq23-29	sq37-78	cp1021-1064	dysprosium	amssq47-62	cp3003-3010	knsq16-22	knsq16-22	sq41-86	cp1037-1068	?.cp4001-4020	sq31-147	knsq1-29	mobile1-35	?.maerlant
2010-04	18.1%		1222										1226	1221						1225	1220			1220	1220
2010-05	23.4%		1290								1323		1302	1294			1322			1301	1292			1288	1314
2010-06	22.6%		1292								2234		2253	2216			2226			2246	2222			2223	2216
2010-07	24.4%		1455								6166		1320	6336			4062			2307	2507			3792	2119
2010-08	1.4%		1004								1007		1008	1005			1017			1006	1005			1024	1320
2010-09	1.3%		1013								1016		1016	1016			1017			1006	1005			1027	1058
2010-10	0.8%		1011								1007		1007	1006			1014			1006	1006			1016	1044
2010-11	2.3%		1039								1022		1021	1023			1032			1022	1021			1033	1049	1004
2010-12	0.7%		1018								1004		1004	1006			1012			1004	1004			1016	1039	1008
2011-01	0.9%		996								1005		1005	1011			1017			1006	1004			1017	1016	1009
2011-02	0.2%		976								1001		999	1000			1007			1001	1004			1000	1002	1006
2011-03	0.3%		970								1002		999	1001			1009			1002	1002			997	1002	1009
2011-04	0.3%		968								1003		1000	1003			1004			1003	1004			995	1003	1008
2011-05	0.5%		964								1003		1000	1006			1002			1003	1009			993	1003	1008
2011-06	18.8%		1278								1235		1269	1226			1246			1253	1214			1240	1231	1240
2011-07	0.6%		971								1004		1000	1006			1004			1005	1010			996	1005	1009	237096
2011-08	6.3%		1052								1063		1062	1070			1063			1063	1071			1054	1065	1080	302258
2011-09	17.7%		1328								1209		1207	1221			1208			1209	1219			1202	1210	1260	406349
2011-10	6.2%		1342			1209		21958	28563		1061		1063	1069			1062			1062	1066			1062	1060
2011-11	0.7%		1220			1019	29847	20005	23643		1005		1005	1010			1004			1004	1009			1003	1004
2011-12	0.4%		1200			26634063	23631	21049	19346		1002		1001	1009			1001			1002	1005			1000	1002		3544
2012-01	0.7%	1094	1184			1008	18341	10596	16153		1003		1004	1011			1003			1003	1010			1002	1002		8291
2012-02	1.3%	1020	1270			1021	12863	11167	11723	1009	1008		1008	1038			1008			1008	1011			1042	1157		16466
2012-03	1.0%	1020	1257			1019	11114	8310	6850	1016	1005		1006	1044	1196		1005			1005	1011			1066			22101
2012-04	4.9%	1002	1185			1246	28518	14158	10331	1002	1004		1003	1041	1516		1002			1002	1009						24366
2012-05	1.5%	1003	591			1001	54716	23612	13090	1001	1001		1001	1042	1168		1001			1002	1016						25579
2012-06	0.4%	1001	629			1014	37010	26109	14885	1001	1002		1001	1053	1196		1002			1001	1008						25497
2012-07	14.7%	1201	592	1010		1012	65623	57994	33362	1140	1091		1038	1054	1228		1156			1091	1244						25547
2012-08	3.2%	1194	563	1020		1019	95813	84078	53305	1212	1007		1007	1056	1251		1006			1007	1008
2012-09	0.4%	1001	613	1013		1015	126317	113675	70828	1000	1001		1001	1058	1040		1001			1001	1006
2012-10	3.7%	1000	560	1016		1016	125048	152557	71502	1001	1001		1001	1049	1152		1001	90		1001	1085
2012-11	0.4%	1000	497	1018		1018	27267	102465	2041	1001	1001		1001	1059	1011		1000	1000		928	1227
2012-12	0.4%	1001	492	1020		255664806	12302	238049	4131	1000	1001		1001	1061	1012		1001	472		1001	1239
2013-01	0.6%	1004	492	1006		1004	7656	425056	2397	1005	1005		1005	1065	1005		1005	845		1019	1239
2013-02	0.0%	1001	562	1001		13602	8757	228542	2441	1001	1001	1002	1003	1057	1000		1001	978	1023	1022	1230
2013-03	0.3%	1003	547	1002		1003	11223	355509	2526	1002	1004	1005		1050	1003		1004	999	1027		1239						626
2013-04	1.2%	1001	456	1023		1001				1001	1001	1002		1051	1002	1000	1082	1002	1095		1227						678
2013-05	0.5%	1001	478	1001		1240	1380	103758	1133	1001	1001	1096		1056	1001	1000	1197	1001	1177		1218	826
2013-06	0.1%	1001	426	1010	868	1001	2594	124849	1069	1001	1001	1134		1057	1000	1000	1173	1000	1200		1202	344
2013-07	4.5%	1000	334	1047	1105	1211				1000	1000			1046	1156	1205	1189	1000			1232	988					835
2013-08	3.8%	1000	401	1000	1001					1000	1001			1032	1146	1223	1180	1000			1209	1135
2013-09	3.8%	1001		1002	1002					1001	1001			1041	1151	1330	1178	1001			1227	1173
2013-10	3.6%	1001		1001	1001					1001	1001			1051	1125	1216	1195	1000			1215	993	1031
2013-11	2.0%	1071		1002	998					1079	1058			1082	1001		1049	1002			1218	994	1136
2013-12	0.4%	1122		1001	989					1116	1082			1105	1000		1014	1000			1210	991	1224
2014-01	1.0%			1011	987										1012		1012	1002				1004	1242
2014-02	15.5%			1001	988		1003	1008	914						1390		1018	1002				994	6974
2014-03	0.5%			1001	981		1002	1006	853						1001		1015	1001				992	1008
Traffic volume: average events per squid per active hour in 1:1000 sampled log
		api	api	mobile	mobile	mobile	ssl	ssl	ssl	text	text	text	text	text	upload	upload	upload	upload	upload	upload	upload	varnish	role?	RIP1	RIP2	RIP3	RIP5
		eqiad	pmpta	eqiad	esams	pmpta	eqiad	esams	pmpta	eqiad	esams	esams	knams	pmpta	eqiad	eqiad	esams	esams	esams	knams	pmpta	eqiad	loc	pmpta	knams	pmpta	loc
month		cp1001-1005	sq33-36	cp1041-1060	cp3011-3014	cp1041-1044	ssl1001-1009	ssl3001-3004	ssl1-4	cp1006-1020	amssq31-46	knsq23-29	knsq23-29	sq37-78	cp1021-1064	dysprosium	amssq47-62	cp3003-3010	knsq16-22	knsq16-22	sq41-86	cp1037-1068	?.cp4001-4020	sq31-147	knsq1-29	mobile1-35	?.maerlant
2010-04			1064										2791	1568						2380	2202			1215	2377
2010-05			1079								2102		1364	1686			1198			1729	2142			1241	1327
2010-06			1224								1946		640	1648			1065			1011	1897			1233	660
2010-07			115								2318		763	3547			1198			1122	2276			1233	669
2010-08			1459								1171		1092	2140			1444			1488	2461			1343	785
2010-09			1464								1311		1313	2341			1548			1617	2458			1425	985
2010-10			1376								1679		1676	2567			1463			1536	2499			1542	1100
2010-11			1552								1878		1887	2781			1612			1693	2538			1649	1238	292
2010-12			1487								1877		1894	2642			1559			1645	2255			1544	1215	305
2011-01			1579								2113		2110	2697			1825			1915	2330			1654	1571	365
2011-02			1343								1844		1842	2589			1796			1885	2533			1701	1610	375
2011-03			1519								1451		1443	2349			1874			1968	2667			1766	1620	343
2011-04			1383								1253		1247	2141			1782			1788	2581			1654	1460	397
2011-05			1431								1334		1326	2129			1820			1821	2532			1700	1492	411
2011-06			1219								1274		1263	2118			1807			1806	2376			1583	1471	307
2011-07			1391								1206		1189	2060			1640			1638	2311			1574	1346	345	8
2011-08			1285								1115		1110	2213			1503			1495	2407			1701	1225	342	68
2011-09			7								1138		1131	2142			1632			1630	2382			1669	1361	269	80
2011-10			4			1059		22	54		1399		1394	2494			2002			2000	2938			1923	1668
2011-11			4			1379	196	103	105		1579		1576	2670			2384			2378	3275			2084	1984
2011-12			4			1689	78	97	95		1505		1489	2448			2369			2362	2935			1946	1972		5
2012-01		9	4			1052	27	47	55		1547		1493	2431			2632			2622	3133			1853	2184		9
2012-02		2014	4			900	47	50	46	2183	1431		1405	734			2849			2766	3469			581	413		12
2012-03		2279	4			968	77	67	48	2284	1427		1313	27	164		2860			2695	3432			22			11
2012-04		2268	4			1139	167	98	65	2278	1355		1455	48	2087		3026			2847	2592						6
2012-05		2675	1			1544	187	156	86	2696	1665		1657	28	1458		3092			3029	2976						5
2012-06		2293	1			1559	156	158	96	2305	1367		1369	21	2		2855			2867	3438						5
2012-07		2236	1	1236		1257	131	163	77	2249	1326		1323	20	2		2525			2515	3307						4
2012-08		2244	1	1277		1281	125	182	80	2249	1272		1270	114	3		2562			2551	3620
2012-09		2488	1	1349		1339	149	225	75	2494	1519		1517	20	749		3082			3040	3686
2012-10		2633	1	1369		1367	180	262	68	2639	1636		1632	41	2197		3338	1		3224	2878
2012-11		2719	1	1263		1267	407	627	4	2718	1805		1804	19	6433		4203	4836		2833	4
2012-12		2449	1	1425		1711	517	1190	4	2450	1677		1678	18	5785		3487	1		3213	3
2013-01		2797	1	1501		1506	749	1487	7	2796	1998		1999	24	6609		4056	7871		3174	10
2013-02		2894	1	1903		2065	859	1287	5	2895	1908	1904	1906	19	6766		2909	10526	2612	2447	3
2013-03		2822	1	2155		2153	1065	1608	6	2822	1842	1844		23	7152		2998	11174	2548		3						1
2013-04		2994	1	2074		2400				2992	1871	1798		21	4983	37262	1186	9779	1027		2						1
2013-05		2828	1	1726		1724	4958	4235	8	2828	2180	517		18	4621	37135	2	10539	2		2	1076
2013-06		2782	1	1358	3	1535	5426	4314	9	2784	2222	8		19	4381	35820	2	9407	2		2	1
2013-07		3295	1	2363	1221	26				3295	2357			25	4727	454	2	8309			2	5					1
2013-08		3547	1	2433	2361					3545	2706			36	4493	2	2	8141			2	695
2013-09		3291		2498	2685					3290	3224			25	4663	2	2	10888			2	672
2013-10		2485		2335	2701					2485	3533			28	5964	2	2	11081			2	3571	428
2013-11		655		2134	2741					655	772			16	12391		3517	11526			2	6869	1028
2013-12		9		2430	2861					9	13			10	12063		3510	10734			2	7262	3
2014-01				2746	3292										12473		3706	11994				7073	3
2014-02				1840	3177		523	2406	1						10521		3676	12222				5247	1631
2014-03				1674	3195		515	2504	1						9380		3584	11987				4700	1778

Calculation of msg loss percentage:

Assume total volume of messages per hour per server is v
We expect average gap between messages per server is 1000
We see avarage gap is g

Msg received percentage is 100 x actual volume / expected volume
Msg loss percentage = 100 - msg rcvd percentage

Actual volume = v / g
Expected volume = v / 1000

Msg rcvd ratio = actual volume / expected volume = (v/g)/(v/1000) = (v/g) x (1000/v)=1000/g
Msg loss perc = 100 - 100 x msg rcvd ratio = 100 - 100 (1000/g)

Examples:
g = 1000 => msg loss perc = 100 - 100 (1000/1000) = 0 %
g = 2000 => msg loss perc = 100 - 100 (1000/2000) = 50 %

g = 1050 => msg loss perc = 100 - 100 (1000/1050) = 4.7%
g = 1100 => msg loss perc = 100 - 100 (1000/1100) = 9.1%