Ingredients Analysis Quality Evaluation - March 2020 - May 2020

From Open Food Facts EN
Jump to navigation Jump to search

In early March 2020, we made an evaluation of the quality of Ingredients Extraction and Analysis so that we could measure the improvements made as part of the Ingredients analysis and search features extraction task of the Project:Personalized_Search.

The metrics have also been recorded in mid May 2020 in order to measure the impact of the improvements we made.

Ingredients parsing and recognition

For each product, the ingredients list is parsed to separate each ingredient. Ingredients that we can match to our multilingual ingredients taxonomy as marked as "known", others as "unknown".

There are different reasons an ingredient can be marked as unknown:

  • The input ingredient list is incorrect (mispellings etc.)
  • The ingredients list contains other things than ingredients and we have not been able to split it at the right place.
  • We have not been able to parse a particular sentence structure or formating (e.g. "a drop of delicious honey with a shower of powder sugar")
  • The ingredient is not yet present in our taxonomy (or not with the right synonym or in the right language)

In the table below,

  • the first column of numbers corresponds to the number of unique ingredients accross all products.
  • And the second column of number corresponds to the number of occurrences of those ingredients. e.g. if a specific ingredient appears in 5 products, it is counted 5 times.

Retrieved on 2020-03-05:

EN - English - UK

2020-03-05 2020-05-14
All products:
Type	Unique tags	Occurrences
known	1702 (8.77%)	104818 (79.85%)
unknown	17706 (91.23%)	26448 (20.15%)
all	19409 (100.00%)	131266 (100.00%)
Type	Unique tags	Occurrences
known	1787 (8.17%)	124626 (80.82%)
unknown	20099 (91.83%)	29578 (19.18%)
all	21886 (100.00%)	154204 (100.00%)
Top 10k most scanned products:
known	1012 (20.67%)	19061 (79.38%)
unknown	3884 (79.31%)	4952 (20.62%)
all	4897 (100.00%)	24013 (100.00%)
known	1055 (22.09%)	21216 (81.93%)
unknown	3720 (77.91%)	4678 (18.07%)
all	4775 (100.00%)	25894 (100.00%)

EN - English - US

Note: ~300K US products were imported from the USDA in April, so results may not be very comparable.

2020-03-21 2020-05-14
All products:
Type	Unique tags	Occurrences
known	2061 (2.12%)	2677857 (82.12%)
unknown	95216 (97.88%)	583085 (17.88%)
all	97278 (100.00%)	3260942 (100.00%)
Type	Unique tags	Occurrences
known	2218 (1.79%)	6457065 (84.67%)
unknown	122007 (98.21%)	1168962 (15.33%)
all	124225 (100.00%)	7626027 (100.00%)
Top 10k most scanned products:
known	983 (18.16%)	25484 (77.58%)
unknown	4430 (81.82%)	7364 (22.42%)
all	5414 (100.00%)	32848 (100.00%)
known	1055 (20.52%)	38068 (82.02%)
unknown	4086 (79.48%)	8347 (17.98%)
all	5141 (100.00%)	46415 (100.00%)

DE - German - Germany

2020-03-05 2020-05-14
All products:
Type	Unique tags	Occurrences
known	2139 (5.36%)	363967 (84.75%)
unknown	37790 (94.64%)	65503 (15.25%)
all	39930 (100.00%)	429470 (100.00%)
Type	Unique tags	Occurrences
known	2233 (5.62%)	417604 (86.72%)
unknown	37522 (94.38%)	63935 (13.28%)
all	39755 (100.00%)	481539 (100.00%)
Top 10k most scanned products:
known	1435 (14.73%)	69066 (85.53%)
unknown	8309 (85.26%)	11689 (14.47%)
all	9745 (100.00%)	80755 (100.00%)
known	1511 (16.88%)	75653 (88.13%)
unknown	7439 (83.12%)	10187 (11.87%)
all	8950 (100.00%)	85840 (100.00%)

ES - Spanish - Spain

2020-03-05 2020-05-14
All products:
Type	Unique tags	Occurrences
known	1970 (4.25%)	176101 (71.95%)
unknown	44352 (95.75%)	68659 (28.05%)
all	46323 (100.00%)	244760 (100.00%)
Type	Unique tags	Occurrences
known	2059 (4.29%)	230207 (76.47%)
unknown	45977 (95.71%)	70846 (23.53%)
all	48036 (100.00%)	301053 (100.00%)
Top 10k most scanned products:
known	1262 (10.47%)	48797 (75.99%)
unknown	10793 (89.52%)	15417 (24.01%)
all	12056 (100.00%)	64214 (100.00%)
known	1354 (12.19%)	55402 (80.84%)
unknown	9751 (87.81%)	13134 (19.16%)
all	11105 (100.00%)	68536 (100.00%)

FI - Finnish - FI

2020-03-05 2020-05-14
All products:
Type	Unique tags	Occurrences
known	829 (43.96%)	11069 (89.15%)
unknown	1056 (55.99%)	1347 (10.85%)
all	1886 (100.00%)	12416 (100.00%)

Type	Unique tags	Occurrences
known	910 (45.09%)	15650 (91.43%)
unknown	1108 (54.91%)	1466 (8.57%)
all	2018 (100.00%)	17116 (100.00%)
Top 10k most scanned products:
known	566 (61.66%)	3465 (89.07%)
unknown	351 (38.24%)	425 (10.93%)
all	918 (100.00%)	3890 (100.00%)
known	581 (67.40%)	3694 (91.96%)
unknown	281 (32.60%)	323 (8.04%)
all	862 (100.00%)	4017 (100.00%)

FR - French - France

2020-03-05 2020-05-14
All products:
Type	Unique tags	Occurrences
known	3825 (0.85%)	3708820 (83.40%)
unknown	446838 (99.15%)	738291 (16.60%)
all	450664 (100.00%)	4447111 (100.00%)
Type	Unique tags	Occurrences
known	3793 (0.92%)	3832574 (85.55%)
unknown	406901 (99.08%)	647239 (14.45%)
all	410694 (100.00%)	4479813 (100.00%)
Top 10k most scanned products:
known	2208 (22.34%)	184218 (94.52%)
unknown	7673 (77.65%)	10686 (5.48%)
all	9882 (100.00%)	194904 (100.00%)
known	2260 (24.29%)	186642 (95.08%)
unknown	7044 (75.71%)	9655 (4.92%)
all	9304 (100.00%)	196297 (100.00%)

IT - Italian - Italy

2020-03-05 2020-05-14
All products:
Type	Unique tags	Occurrences
known	1319 (9.43%)	53345 (74.58%)
unknown	12669 (90.56%)	18186 (25.42%)
all	13989 (100.00%)	71531 (100.00%)
Type	Unique tags	Occurrences
known	1418 (9.21%)	67900 (78.54%)
unknown	13976 (90.79%)	18557 (21.46%)
all	15394 (100.00%)	86457 (100.00%)
Top 10k most scanned products:
known	911 (13.11%)	20838 (72.21%)
unknown	6039 (86.88%)	8018 (27.79%)
all	6951 (100.00%)	28856 (100.00%)
known	964 (14.90%)	24072 (78.11%)
unknown	5507 (85.10%)	6747 (21.89%)
all	6471 (100.00%)	30819 (100.00%)

NL - Dutch - The Netherlands

2020-03-05 2020-05-14
All products:
Type	Unique tags	Occurrences
known	1304 (17.03%)	31491 (78.84%)
unknown	6353 (82.96%)	8453 (21.16%)
all	7658 (100.00%)	39944 (100.00%)
Type	Unique tags	Occurrences
known	1391 (17.72%)	36019 (80.81%)
unknown	6461 (82.28%)	8555 (19.19%)
all	7852 (100.00%)	44574 (100.00%)
Top 10k most scanned products:
known	755 (24.25%)	8069 (73.71%)
unknown	2358 (75.72%)	2878 (26.29%)
all	3114 (100.00%)	10947 (100.00%)
known	797 (29.00%)	9104 (79.28%)
unknown	1951 (71.00%)	2379 (20.72%)
all	2748 (100.00%)	11483 (100.00%)

PT - Portuguese - Portugal

2020-03-05 2020-05-14
All products:
Type	Unique tags	Occurrences
known	918 (16.05%)	15816 (70.50%)
unknown	4800 (83.93%)	6617 (29.50%)
all	5719 (100.00%)	22433 (100.00%)

Type	Unique tags	Occurrences
known	948 (16.10%)	17005 (71.53%)
unknown	4940 (83.90%)	6769 (28.47%)
all	5888 (100.00%)	23774 (100.00%)

Top 10k most scanned products:
known	578 (19.37%)	6261 (68.40%)
unknown	2405 (80.60%)	2892 (31.60%)
all	2984 (100.00%)	9153 (100.00%)
known	602 (20.40%)	6608 (70.43%)
unknown	2349 (79.60%)	2775 (29.57%)
all	2951 (100.00%)	9383 (100.00%)

Ingredients analysis

For each product, once we have parsed the ingredients list to separate each ingredient and match them to our taxonomy, we can analyze if the products are vegan, vegetarian, without palm oil etc.

To mark a product as non-vegan etc. we only need to find one non-vegan ingredient. But to mark a product as vegan, we need to be sure that we have been able to correctly recognize all ingredients so that we can verify that all of them are vegan.

Retrieved on 2020-03-05:

EN - English - UK

2020-03-05 2020-05-14
All products:
Vegetarian status unknown	4814	
Palm oil content unknown	4138	
Vegan status unknown	3281	
Non-vegan	2525	
Palm oil free	1286	
Vegetarian	903	
Palm oil	875	
Vegan	725	
Non-vegetarian	719	
May contain palm oil	314	
Maybe vegetarian	177	
Maybe vegan	82	
Vegetarian status unknown	4533	
Palm oil content unknown	4513	
Vegan status unknown	3251	
Non-vegan	2919	
Vegetarian	2785	
Vegan	2043	
Palm oil free	1587	
Palm oil	1049	
Non-vegetarian	853	
May contain palm oil	380	
Maybe vegetarian	214	
Maybe vegan	102	
Top 10k most scanned products:
Vegetarian status unknown	1016	
Palm oil content unknown	817	
Vegan status unknown	711	
Non-vegan	439	
Palm oil free	222	
Palm oil	192	
Vegetarian	171	
Vegan	139	
Non-vegetarian	81	
May contain palm oil	73	
Maybe vegetarian	36	
Maybe vegan	15	
Vegetarian status unknown	835	
Palm oil content unknown	816	
Vegan status unknown	616	
Non-vegan	485	
Vegetarian	470	
Vegan	314	
Palm oil free	264	
Palm oil	219	
Non-vegetarian	92	
May contain palm oil	80	
Maybe vegetarian	45	
Maybe vegan	18	


DE - German - Germany

2020-03-05 2020-05-14
All products:
Vegetarian status unknown	15605	
Palm oil content unknown	14599	
Vegan status unknown	10377	
Non-vegan	10151	
Palm oil free	6199	
Vegetarian	4394	
Vegan	3367	
Non-vegetarian	2805	
Palm oil	2357	
Maybe vegetarian	1486	
May contain palm oil	1135	
Maybe vegan	395	
Palm oil content unknown	15370	
Vegetarian status unknown	14019	
Non-vegan	11373	
Vegetarian	9087	
Vegan status unknown	8791	
Palm oil free	7765	
Vegan	7377	
Non-vegetarian	3225	
Palm oil	2592	
Maybe vegetarian	1731	
May contain palm oil	1365	
Maybe vegan	462	
Top 10k most scanned products:
Vegetarian status unknown	2925	
Palm oil content unknown	2449	
Non-vegan	1903	
Vegan status unknown	1874	
Palm oil free	1255	
Vegetarian	849	
Vegan	634	
Palm oil	567	
Maybe vegetarian	381	
Non-vegetarian	362	
May contain palm oil	246	
Maybe vegan	106	
Palm oil content unknown	2386	
Vegetarian status unknown	2384	
Non-vegan	2030	
Vegetarian	1626	
Palm oil free	1471	
Vegan status unknown	1419	
Vegan	1249	
Palm oil	600	
Maybe vegetarian	420	
Non-vegetarian	397	
May contain palm oil	279	
Maybe vegan	118	

ES - Spanish - Spain

2020-03-05 2020-05-14
All products:
Vegetarian status unknown	12076	
Palm oil content unknown	11192	
Vegan status unknown	9812	
Non-vegan	3877	
Palm oil free	2707	
Vegetarian	1906	
Vegan	1689	
Non-vegetarian	1145	
May contain palm oil	978	
Palm oil	809	
Maybe vegetarian	559	
Maybe vegan	308	
Palm oil content unknown	12221	
Vegetarian status unknown	10298	
Vegetarian	8136	
Vegan	7605	
Vegan status unknown	7515	
Non-vegan	5481	
Palm oil free	4988	
Non-vegetarian	1805	
May contain palm oil	1171	
Palm oil	1022	
Maybe vegetarian	664	
Maybe vegan	286	
Top 10k most scanned products:
Vegetarian status unknown	2733	
Palm oil content unknown	2387	
Vegan status unknown	2081	
Non-vegan	1105	
Palm oil free	733	
Vegetarian	484	
Vegan	394	
May contain palm oil	308	
Non-vegetarian	272	
Palm oil	267	
Maybe vegetarian	206	
Maybe vegan	115	
Palm oil content unknown	2235	
Vegetarian status unknown	2216	
Vegan status unknown	1540	
Non-vegan	1316	
Vegetarian	1096	
Palm oil free	997	
Vegan	925	
Non-vegetarian	362	
May contain palm oil	328	
Palm oil	286	
Maybe vegetarian	209	
Maybe vegan	100	

FI - Finnish - FI

2020-03-05 2020-05-14
All products:
Vegetarian status unknown	375	
Palm oil content unknown	312	
Non-vegan	278	
Vegan status unknown	245	
Palm oil free	191	
Maybe vegetarian	94	
Non-vegetarian	91	
Palm oil	85	
Vegetarian	83	
Vegan	76	
May contain palm oil	55	
Maybe vegan	44	
Vegetarian status unknown	433	
Palm oil content unknown	396	
Non-vegan	364	
Vegan status unknown	283	
Palm oil free	256	
Vegetarian	178	
Vegan	153	
Non-vegetarian	128	
Maybe vegetarian	115	
Palm oil	112	
May contain palm oil	78	
Maybe vegan	52	
Top 10k most scanned products:
Vegetarian status unknown	101	
Palm oil content unknown	93	
Non-vegan	77	
Vegan status unknown	70	
Palm oil free	59	
Non-vegetarian	31	
Vegetarian	27	
Maybe vegetarian	25	
Vegan	24	
Palm oil	19	
Maybe vegan	13	
May contain palm oil	13	
Vegetarian status unknown	88	
Palm oil content unknown	86	
Non-vegan	79	
Palm oil free	72	
Vegan status unknown	58	
Vegetarian	46	
Vegan	41	
Non-vegetarian	31	
Maybe vegetarian	28	
Palm oil	22	
Maybe vegan	15	
May contain palm oil	13	

FR - French - France

2020-03-05 2020-05-14
All products:
Palm oil content unknown	128788	
Vegetarian status unknown	127223	
Non-vegan	105765	
Vegan status unknown	82754	
Palm oil free	66322	
Non-vegetarian	42135	
Vegetarian	36283	
Vegan	27356	
Maybe vegetarian	16327	
Palm oil	14315	
May contain palm oil	12543	
Maybe vegan	6093	
Palm oil content unknown	120790	
Vegetarian status unknown	118197	
Non-vegan	107363	
Palm oil free	77233	
Vegan status unknown	75502	
Vegetarian	50863	
Non-vegetarian	42829	
Vegan	39716	
Maybe vegetarian	17543	
Palm oil	14895	
May contain palm oil	12870	
Maybe vegan	6712	
Top 10k most scanned products:
Non-vegan	4629	
Vegetarian status unknown	4489	
Palm oil free	4277	
Palm oil content unknown	3643	
Vegan status unknown	2542	
Vegetarian	2420	
Vegan	1723	
Maybe vegetarian	1460	
Non-vegetarian	1089	
Palm oil	897	
May contain palm oil	641	
Maybe vegan	564	
Palm oil free	4895	
Non-vegan	4607	
Vegetarian status unknown	4029	
Palm oil content unknown	3089	
Vegetarian	2827	
Vegan status unknown	2302	
Vegan	1944	
Maybe vegetarian	1523	
Non-vegetarian	1082	
Palm oil	889	
Maybe vegan	605	
May contain palm oil	594	

IT - Italian - Italy

2020-03-05 2020-05-14
All products:
Vegetarian status unknown	3226	
Palm oil content unknown	2965	
Vegan status unknown	2229	
Non-vegan	1492	
Palm oil free	748	
Vegetarian	507	
Vegan	412	
Non-vegetarian	295	
Palm oil	279	
May contain palm oil	208	
Maybe vegetarian	172	
Maybe vegan	67	
Vegetarian status unknown	3357	
Palm oil content unknown	3055	
Vegan status unknown	2267	
Non-vegan	1820	
Palm oil free	1498	
Vegetarian	951	
Vegan	770	
Non-vegetarian	382	
Palm oil	316	
Maybe vegetarian	272	
May contain palm oil	252	
Maybe vegan	102	
Top 10k most scanned products:
Vegetarian status unknown	1320	
Palm oil content unknown	1206	
Vegan status unknown	932	
Non-vegan	578	
Palm oil free	290	
Vegetarian	205	
Vegan	159	
Palm oil	118	
Non-vegetarian	107	
May contain palm oil	81	
Maybe vegetarian	63	
Maybe vegan	26	
Vegetarian status unknown	1230	
Palm oil content unknown	1063	
Vegan status unknown	839	
Non-vegan	637	
Palm oil free	572	
Vegetarian	343	
Vegan	268	
Palm oil	123	
Non-vegetarian	120	
Maybe vegetarian	91	
May contain palm oil	89	
Maybe vegan	37	

NL - Dutch - The Netherlands

2020-03-05 2020-05-14
All products:
Vegetarian status unknown	1663	
Palm oil content unknown	1442	
Vegan status unknown	1202	
Non-vegan	729	
Palm oil free	504	
Vegetarian	382	
Vegan	322	
Palm oil	229	
Non-vegetarian	161	
May contain palm oil	119	
Maybe vegetarian	88	
Maybe vegan	41	
Palm oil content unknown	1506	
Vegetarian status unknown	1477	
Vegetarian	1066	
Vegan status unknown	1027	
Vegan	921	
Non-vegan	826	
Palm oil free	634	
Palm oil	251	
Non-vegetarian	187	
May contain palm oil	129	
Maybe vegetarian	91	
Maybe vegan	44	
Top 10k most scanned products:
Vegetarian status unknown	447	
Palm oil content unknown	364	
Vegan status unknown	319	
Non-vegan	189	
Palm oil free	113	
Vegetarian	79	
Vegan	67	
Palm oil	66	
May contain palm oil	42	
Non-vegetarian	36	
Maybe vegetarian	23	
Maybe vegan	10	
Vegetarian status unknown	402	
Palm oil content unknown	366	
Vegan status unknown	274	
Non-vegan	209	
Vegetarian	160	
Palm oil free	139	
Vegan	133	
Palm oil	70	
Non-vegetarian	41	
May contain palm oil	39	
Maybe vegetarian	26	
Maybe vegan	11	

PT - Portuguese - Portugal

2020-03-05 2020-05-14
All products:
Vegetarian status unknown	1072	
Palm oil content unknown	914	
Vegan status unknown	768	
Non-vegan	454	
Palm oil free	187	
May contain palm oil	140	
Vegetarian	126	
Palm oil	105	
Vegan	104	
Non-vegetarian	97	
Maybe vegetarian	51	
Maybe vegan	20	
Vegetarian status unknown	1000	
Palm oil content unknown	909	
Vegan status unknown	700	
Non-vegan	478	
Vegetarian	265	
Palm oil free	245	
Vegan	228	
May contain palm oil	138	
Palm oil	112	
Non-vegetarian	108	
Maybe vegetarian	56	
Maybe vegan	23	
Top 10k most scanned products:
Vegetarian status unknown	453	
Palm oil content unknown	375	
Vegan status unknown	328	
Non-vegan	171	
Palm oil free	70	
May contain palm oil	53	
Vegetarian	48	
Palm oil	45	
Vegan	36	
Non-vegetarian	22	
Maybe vegetarian	20	
Maybe vegan	8	
Vegetarian status unknown	411	
Palm oil content unknown	357	
Vegan status unknown	288	
Non-vegan	177	
Vegetarian	104	
Palm oil free	95	
Vegan	85	
May contain palm oil	49	
Palm oil	48	
Maybe vegetarian	24	
Non-vegetarian	22	
Maybe vegan	11	


Evolution of the metrics between early March 2020 and mid May 2020

Comparison of key metrics

The comparison table contains the metrics for all products of a country.

Metrics:

  • Ingredients parsing and recognition: % of known ingredients for all ingredients occurences.
  • Ingredients analysis: known vegetarian status
    • = 1 - "Vegetarian status unknown" / ("Vegetarian status unknown" + "Vegetarian" + "Non vegetarian" + "Maybe vegetarian")
Country % known ingredients March 2020 % known ingredients May 2020 % known vegetarian status March 2020 % known vegetarian status May 2020
UK 79.75% 80.82% 27.20% 45.94%
US 82.12% 84.67% not recorded not recorded
DE 84.75% 86.72% 35.76% 50.04%
ES 71.95% 76.47% 23.01% 50.73%
FI 89.15% 91.43% 41.68% 49.30%
FR 83.40% 85.55% 42.68% 48.47%
IT 74.58% 78.54% 23.19% 32.35%
NL 78.84% 80.81% 27.51% 47.64%
PT 70.50% 71.53% 20.36% 26.74%

Observations

Significant progress has been made on all tested countries.

For ingredients parsing and recognition, it becomes harder and harder to get big increases in the % of known ingredients. It is expected as we have focused first on the improvements that had the biggest impact. But even a "small" increase from 83.40% to 85.55% for known ingredients in France corresponds to a drop of the unknown ingredients from 16.60% to 14.65% which is a 11.7% relative drop.

It is very interesting to see that the improvements in ingredients parsing and recognition translate in much more significant ingredients analysis (knowing the vegetarian status of a product) improvements. All countries had a 5% to 25% absolute increase of products with known vegetarian status.