Difference between revisions of "Ingredients Analysis Quality Evaluation - March 2020 - May 2020"
Line 53: | Line 53: | ||
=== EN - English - US === | === EN - English - US === | ||
− | = | + | {| class="wikitable" |
− | + | |- | |
− | + | ! !! 2020-03-21 !! 2020-05-14 | |
− | + | |- | |
+ | | [https://us.openfoodfacts.org/ingredients?stats=1 All products:] || | ||
<pre> | <pre> | ||
known 2061 (2.12%) 2677857 (82.12%) | known 2061 (2.12%) 2677857 (82.12%) | ||
Line 62: | Line 63: | ||
all 97278 (100.00%) 3260942 (100.00%) | all 97278 (100.00%) 3260942 (100.00%) | ||
</pre> | </pre> | ||
+ | || | ||
+ | <pre> | ||
− | + | </pre> | |
− | + | |- | |
− | + | | [https://us.openfoodfacts.org/popularity/top-10000-us-scans-2019/ingredients?stats=1 Top 10k most scanned products:] || | |
− | |||
<pre> | <pre> | ||
known 983 (18.16%) 25484 (77.58%) | known 983 (18.16%) 25484 (77.58%) | ||
Line 72: | Line 74: | ||
all 5414 (100.00%) 32848 (100.00%) | all 5414 (100.00%) 32848 (100.00%) | ||
</pre> | </pre> | ||
+ | || | ||
+ | <pre> | ||
+ | |||
+ | </pre> | ||
+ | |} | ||
+ | |||
=== DE - German - Germany === | === DE - German - Germany === | ||
− | [https://de.openfoodfacts.org/ingredients?stats=1 All products:] | + | {| class="wikitable" |
− | + | |- | |
+ | ! !! 2020-03-05 !! 2020-05-14 | ||
+ | |- | ||
+ | | [https://de.openfoodfacts.org/ingredients?stats=1 All products:] || | ||
<pre> | <pre> | ||
known 2139 (5.36%) 363967 (84.75%) | known 2139 (5.36%) 363967 (84.75%) | ||
Line 82: | Line 93: | ||
all 39930 (100.00%) 429470 (100.00%) | all 39930 (100.00%) 429470 (100.00%) | ||
</pre> | </pre> | ||
+ | || | ||
+ | <pre> | ||
− | [https://de.openfoodfacts.org/popularity/top-10000-de-scans-2019/ingredients?stats=1 Top 10k most scanned products:] | + | </pre> |
− | + | |- | |
+ | | [https://de.openfoodfacts.org/popularity/top-10000-de-scans-2019/ingredients?stats=1 Top 10k most scanned products:] || | ||
<pre> | <pre> | ||
known 1435 (14.73%) 69066 (85.53%) | known 1435 (14.73%) 69066 (85.53%) | ||
Line 90: | Line 104: | ||
all 9745 (100.00%) 80755 (100.00%) | all 9745 (100.00%) 80755 (100.00%) | ||
</pre> | </pre> | ||
+ | || | ||
+ | <pre> | ||
+ | |||
+ | </pre> | ||
+ | |} | ||
+ | |||
+ | |||
+ | |||
=== ES - Spanish - Spain === | === ES - Spanish - Spain === | ||
− | [https://es.openfoodfacts.org/ingredients?stats=1 All products:] | + | {| class="wikitable" |
− | + | |- | |
+ | ! !! 2020-03-05 !! 2020-05-14 | ||
+ | |- | ||
+ | | [https://es.openfoodfacts.org/ingredients?stats=1 All products:] || | ||
<pre> | <pre> | ||
known 1970 (4.25%) 176101 (71.95%) | known 1970 (4.25%) 176101 (71.95%) | ||
Line 100: | Line 125: | ||
all 46323 (100.00%) 244760 (100.00%) | all 46323 (100.00%) 244760 (100.00%) | ||
</pre> | </pre> | ||
+ | || | ||
+ | <pre> | ||
− | [https://es.openfoodfacts.org/popularity/top-10000-es-scans-2019/ingredients?stats=1 Top 10k most scanned products:] | + | </pre> |
− | + | |- | |
+ | | [https://es.openfoodfacts.org/popularity/top-10000-es-scans-2019/ingredients?stats=1 Top 10k most scanned products:] || | ||
<pre> | <pre> | ||
known 1262 (10.47%) 48797 (75.99%) | known 1262 (10.47%) 48797 (75.99%) | ||
Line 108: | Line 136: | ||
all 12056 (100.00%) 64214 (100.00%) | all 12056 (100.00%) 64214 (100.00%) | ||
</pre> | </pre> | ||
+ | || | ||
+ | <pre> | ||
+ | |||
+ | </pre> | ||
+ | |} | ||
+ | |||
+ | |||
=== FI - Finnish - FI === | === FI - Finnish - FI === | ||
− | [https://fi.openfoodfacts.org/ingredients?stats=1 All products:] | + | {| class="wikitable" |
− | + | |- | |
+ | ! !! 2020-03-05 !! 2020-05-14 | ||
+ | |- | ||
+ | | [https://fi.openfoodfacts.org/ingredients?stats=1 All products:] || | ||
<pre> | <pre> | ||
known 829 (43.96%) 11069 (89.15%) | known 829 (43.96%) 11069 (89.15%) | ||
Line 118: | Line 156: | ||
all 1886 (100.00%) 12416 (100.00%) | all 1886 (100.00%) 12416 (100.00%) | ||
</pre> | </pre> | ||
+ | || | ||
+ | <pre> | ||
− | [https://fi.openfoodfacts.org/popularity/top-10000-fi-scans-2019/ingredients?stats=1 Top 10k most scanned products:] | + | </pre> |
− | + | |- | |
+ | | [https://fi.openfoodfacts.org/popularity/top-10000-fi-scans-2019/ingredients?stats=1 Top 10k most scanned products:] | ||
+ | || | ||
<pre> | <pre> | ||
known 566 (61.66%) 3465 (89.07%) | known 566 (61.66%) 3465 (89.07%) | ||
Line 126: | Line 168: | ||
all 918 (100.00%) 3890 (100.00%) | all 918 (100.00%) 3890 (100.00%) | ||
</pre> | </pre> | ||
+ | || | ||
+ | <pre> | ||
+ | |||
+ | </pre> | ||
+ | |} | ||
+ | |||
+ | |||
=== FR - French - France === | === FR - French - France === | ||
− | = | + | {| class="wikitable" |
− | + | |- | |
+ | ! !! 2020-03-05 !! 2020-05-14 | ||
+ | |- | ||
+ | | [https://fr.openfoodfacts.org/ingredients?stats=1 All products:] || | ||
<pre> | <pre> | ||
known 3825 (0.85%) 3708820 (83.40%) | known 3825 (0.85%) 3708820 (83.40%) | ||
Line 136: | Line 188: | ||
all 450664 (100.00%) 4447111 (100.00%) | all 450664 (100.00%) 4447111 (100.00%) | ||
</pre> | </pre> | ||
+ | || | ||
+ | <pre> | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
</pre> | </pre> | ||
− | + | |- | |
− | + | | [https://fr.openfoodfacts.org/popularity/top-10000-fr-scans-2019/ingredients?stats=1 Top 10k most scanned products:] || | |
− | |||
<pre> | <pre> | ||
known 2208 (22.34%) 184218 (94.52%) | known 2208 (22.34%) 184218 (94.52%) | ||
Line 154: | Line 199: | ||
all 9882 (100.00%) 194904 (100.00%) | all 9882 (100.00%) 194904 (100.00%) | ||
</pre> | </pre> | ||
+ | || | ||
+ | <pre> | ||
+ | |||
+ | </pre> | ||
+ | |} | ||
+ | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
=== IT - Italian - Italy === | === IT - Italian - Italy === | ||
− | [https://it.openfoodfacts.org/ingredients?stats=1 All products:] | + | {| class="wikitable" |
− | + | |- | |
+ | ! !! 2020-03-05 !! 2020-05-14 | ||
+ | |- | ||
+ | | [https://it.openfoodfacts.org/ingredients?stats=1 All products:] || | ||
<pre> | <pre> | ||
known 1319 (9.43%) 53345 (74.58%) | known 1319 (9.43%) 53345 (74.58%) | ||
Line 172: | Line 220: | ||
all 13989 (100.00%) 71531 (100.00%) | all 13989 (100.00%) 71531 (100.00%) | ||
</pre> | </pre> | ||
+ | || | ||
+ | <pre> | ||
− | [https://it.openfoodfacts.org/popularity/top-10000-it-scans-2019/ingredients?stats=1 Top 10k most scanned products:] | + | </pre> |
− | + | |- | |
+ | | [https://it.openfoodfacts.org/popularity/top-10000-it-scans-2019/ingredients?stats=1 Top 10k most scanned products:] || | ||
<pre> | <pre> | ||
known 911 (13.11%) 20838 (72.21%) | known 911 (13.11%) 20838 (72.21%) | ||
Line 180: | Line 231: | ||
all 6951 (100.00%) 28856 (100.00%) | all 6951 (100.00%) 28856 (100.00%) | ||
</pre> | </pre> | ||
+ | || | ||
+ | <pre> | ||
+ | |||
+ | </pre> | ||
+ | |} | ||
+ | |||
=== NL - Dutch - The Netherlands === | === NL - Dutch - The Netherlands === | ||
− | = | + | {| class="wikitable" |
− | + | |- | |
+ | ! !! 2020-03-05 !! 2020-05-14 | ||
+ | |- | ||
+ | | [https://nl.openfoodfacts.org/ingredients?stats=1 All products:] || | ||
<pre> | <pre> | ||
known 1304 (17.03%) 31491 (78.84%) | known 1304 (17.03%) 31491 (78.84%) | ||
Line 191: | Line 251: | ||
</pre> | </pre> | ||
− | + | || | |
+ | <pre> | ||
− | |||
− | |||
− | |||
− | |||
</pre> | </pre> | ||
− | + | |- | |
− | + | | [https://nl.openfoodfacts.org/popularity/top-10000-nl-scans-2019/ingredients?stats=1 Top 10k most scanned products:] || | |
− | |||
<pre> | <pre> | ||
known 755 (24.25%) 8069 (73.71%) | known 755 (24.25%) 8069 (73.71%) | ||
Line 206: | Line 262: | ||
all 3114 (100.00%) 10947 (100.00%) | all 3114 (100.00%) 10947 (100.00%) | ||
</pre> | </pre> | ||
+ | || | ||
+ | <pre> | ||
− | + | </pre> | |
+ | |} | ||
− | |||
− | |||
− | |||
− | |||
− | |||
=== PT - Portuguese - Portugal === | === PT - Portuguese - Portugal === | ||
− | [https://pt.openfoodfacts.org/ingredients?stats=1 All products:] | + | {| class="wikitable" |
− | + | |- | |
+ | ! !! 2020-03-05 !! 2020-05-14 | ||
+ | |- | ||
+ | | [https://pt.openfoodfacts.org/ingredients?stats=1 All products:] || | ||
<pre> | <pre> | ||
known 918 (16.05%) 15816 (70.50%) | known 918 (16.05%) 15816 (70.50%) | ||
Line 224: | Line 281: | ||
all 5719 (100.00%) 22433 (100.00%) | all 5719 (100.00%) 22433 (100.00%) | ||
</pre> | </pre> | ||
+ | || | ||
+ | <pre> | ||
− | [https://pt.openfoodfacts.org/popularity/top-10000-pt-scans-2019/ingredients?stats=1 Top 10k most scanned products:] | + | </pre> |
− | + | |- | |
+ | | [https://pt.openfoodfacts.org/popularity/top-10000-pt-scans-2019/ingredients?stats=1 Top 10k most scanned products:] || | ||
<pre> | <pre> | ||
known 578 (19.37%) 6261 (68.40%) | known 578 (19.37%) 6261 (68.40%) | ||
Line 232: | Line 292: | ||
all 2984 (100.00%) 9153 (100.00%) | all 2984 (100.00%) 9153 (100.00%) | ||
</pre> | </pre> | ||
+ | || | ||
+ | <pre> | ||
− | + | </pre> | |
− | + | |} | |
== Ingredients analysis == | == Ingredients analysis == |
Revision as of 13:11, 14 May 2020
In early March 2020, we made an evaluation of the quality of ingredients analysis so that we could measure the improvements made as part of the Ingredients analysis and search features extraction task of the Project:Personalized_Search.
Ingredients parsing and recognition
For each product, the ingredients list is parsed to separate each ingredient. Ingredients that we can match to our multilingual ingredients taxonomy as marked as "known", others as "unknown".
There are different reasons an ingredient can be marked as unknown:
- The input ingredient list is incorrect (mispellings etc.)
- The ingredients list contains other things than ingredients and we have not been able to split it at the right place.
- We have not been able to parse a particular sentence structure or formating (e.g. "a drop of delicious honey with a shower of powder sugar")
- The ingredient is not yet present in our taxonomy (or not with the right synonym or in the right language)
In the table below, the first column of numbers corresponds to the number of unique ingredients accross all products. And the second column of number corresponds to the number of occurrences of those ingredients. e.g. if a specific ingredient appears in 5 products, it is counted 5 times.
Retrieved on 2020-03-05:
EN - English - UK
2020-03-05 | 2020-05-14 | |
---|---|---|
All products: |
known 1702 (8.77%) 104818 (79.85%) unknown 17706 (91.23%) 26448 (20.15%) all 19409 (100.00%) 131266 (100.00%) |
Type Unique tags Occurrences known 1787 (8.17%) 124626 (80.82%) unknown 20099 (91.83%) 29578 (19.18%) all 21886 (100.00%) 154204 (100.00%) |
Top 10k most scanned products: |
known 1012 (20.67%) 19061 (79.38%) unknown 3884 (79.31%) 4952 (20.62%) all 4897 (100.00%) 24013 (100.00%) |
known 1055 (22.09%) 21216 (81.93%) unknown 3720 (77.91%) 4678 (18.07%) all 4775 (100.00%) 25894 (100.00%) |
EN - English - US
2020-03-21 | 2020-05-14 | |
---|---|---|
All products: |
known 2061 (2.12%) 2677857 (82.12%) unknown 95216 (97.88%) 583085 (17.88%) all 97278 (100.00%) 3260942 (100.00%) |
|
Top 10k most scanned products: |
known 983 (18.16%) 25484 (77.58%) unknown 4430 (81.82%) 7364 (22.42%) all 5414 (100.00%) 32848 (100.00%) |
|
DE - German - Germany
2020-03-05 | 2020-05-14 | |
---|---|---|
All products: |
known 2139 (5.36%) 363967 (84.75%) unknown 37790 (94.64%) 65503 (15.25%) all 39930 (100.00%) 429470 (100.00%) |
|
Top 10k most scanned products: |
known 1435 (14.73%) 69066 (85.53%) unknown 8309 (85.26%) 11689 (14.47%) all 9745 (100.00%) 80755 (100.00%) |
|
ES - Spanish - Spain
2020-03-05 | 2020-05-14 | |
---|---|---|
All products: |
known 1970 (4.25%) 176101 (71.95%) unknown 44352 (95.75%) 68659 (28.05%) all 46323 (100.00%) 244760 (100.00%) |
|
Top 10k most scanned products: |
known 1262 (10.47%) 48797 (75.99%) unknown 10793 (89.52%) 15417 (24.01%) all 12056 (100.00%) 64214 (100.00%) |
|
FI - Finnish - FI
2020-03-05 | 2020-05-14 | |
---|---|---|
All products: |
known 829 (43.96%) 11069 (89.15%) unknown 1056 (55.99%) 1347 (10.85%) all 1886 (100.00%) 12416 (100.00%) |
|
Top 10k most scanned products: |
known 566 (61.66%) 3465 (89.07%) unknown 351 (38.24%) 425 (10.93%) all 918 (100.00%) 3890 (100.00%) |
|
FR - French - France
2020-03-05 | 2020-05-14 | |
---|---|---|
All products: |
known 3825 (0.85%) 3708820 (83.40%) unknown 446838 (99.15%) 738291 (16.60%) all 450664 (100.00%) 4447111 (100.00%) |
|
Top 10k most scanned products: |
known 2208 (22.34%) 184218 (94.52%) unknown 7673 (77.65%) 10686 (5.48%) all 9882 (100.00%) 194904 (100.00%) |
|
IT - Italian - Italy
2020-03-05 | 2020-05-14 | |
---|---|---|
All products: |
known 1319 (9.43%) 53345 (74.58%) unknown 12669 (90.56%) 18186 (25.42%) all 13989 (100.00%) 71531 (100.00%) |
|
Top 10k most scanned products: |
known 911 (13.11%) 20838 (72.21%) unknown 6039 (86.88%) 8018 (27.79%) all 6951 (100.00%) 28856 (100.00%) |
|
NL - Dutch - The Netherlands
2020-03-05 | 2020-05-14 | |
---|---|---|
All products: |
known 1304 (17.03%) 31491 (78.84%) unknown 6353 (82.96%) 8453 (21.16%) all 7658 (100.00%) 39944 (100.00%) |
|
Top 10k most scanned products: |
known 755 (24.25%) 8069 (73.71%) unknown 2358 (75.72%) 2878 (26.29%) all 3114 (100.00%) 10947 (100.00%) |
|
PT - Portuguese - Portugal
2020-03-05 | 2020-05-14 | |
---|---|---|
All products: |
known 918 (16.05%) 15816 (70.50%) unknown 4800 (83.93%) 6617 (29.50%) all 5719 (100.00%) 22433 (100.00%) |
|
Top 10k most scanned products: |
known 578 (19.37%) 6261 (68.40%) unknown 2405 (80.60%) 2892 (31.60%) all 2984 (100.00%) 9153 (100.00%) |
|
Ingredients analysis
For each product, once we have parsed the ingredients list to separate each ingredient and match them to our taxonomy, we can analyze if the products are vegan, vegetarian, without palm oil etc.
To mark a product as non-vegan etc. we only need to find one non-vegan ingredient. But to mark a product as vegan, we need to be sure that we have been able to correctly recognize all ingredients so that we can verify that all of them are vegan.
Retrieved on 2020-03-05:
EN - English - UK
Vegetarian status unknown 4814 Palm oil content unknown 4138 Vegan status unknown 3281 Non-vegan 2525 Palm oil free 1286 Vegetarian 903 Palm oil 875 Vegan 725 Non-vegetarian 719 May contain palm oil 314 Maybe vegetarian 177 Maybe vegan 82
Top 10k most scanned products:
Vegetarian status unknown 1016 Palm oil content unknown 817 Vegan status unknown 711 Non-vegan 439 Palm oil free 222 Palm oil 192 Vegetarian 171 Vegan 139 Non-vegetarian 81 May contain palm oil 73 Maybe vegetarian 36 Maybe vegan 15
DE - German - Germany
Vegetarian status unknown 15605 Palm oil content unknown 14599 Vegan status unknown 10377 Non-vegan 10151 Palm oil free 6199 Vegetarian 4394 Vegan 3367 Non-vegetarian 2805 Palm oil 2357 Maybe vegetarian 1486 May contain palm oil 1135 Maybe vegan 395
Top 10k most scanned products:
Vegetarian status unknown 2925 Palm oil content unknown 2449 Non-vegan 1903 Vegan status unknown 1874 Palm oil free 1255 Vegetarian 849 Vegan 634 Palm oil 567 Maybe vegetarian 381 Non-vegetarian 362 May contain palm oil 246 Maybe vegan 106
ES - Spanish - Spain
Vegetarian status unknown 12076 Palm oil content unknown 11192 Vegan status unknown 9812 Non-vegan 3877 Palm oil free 2707 Vegetarian 1906 Vegan 1689 Non-vegetarian 1145 May contain palm oil 978 Palm oil 809 Maybe vegetarian 559 Maybe vegan 308
Top 10k most scanned products:
Vegetarian status unknown 2733 Palm oil content unknown 2387 Vegan status unknown 2081 Non-vegan 1105 Palm oil free 733 Vegetarian 484 Vegan 394 May contain palm oil 308 Non-vegetarian 272 Palm oil 267 Maybe vegetarian 206 Maybe vegan 115
FI - Finnish - FI
Vegetarian status unknown 375 Palm oil content unknown 312 Non-vegan 278 Vegan status unknown 245 Palm oil free 191 Maybe vegetarian 94 Non-vegetarian 91 Palm oil 85 Vegetarian 83 Vegan 76 May contain palm oil 55 Maybe vegan 44
Top 10k most scanned products:
Vegetarian status unknown 101 Palm oil content unknown 93 Non-vegan 77 Vegan status unknown 70 Palm oil free 59 Non-vegetarian 31 Vegetarian 27 Maybe vegetarian 25 Vegan 24 Palm oil 19 Maybe vegan 13 May contain palm oil 13
FR - French - France
Palm oil content unknown 128788 Vegetarian status unknown 127223 Non-vegan 105765 Vegan status unknown 82754 Palm oil free 66322 Non-vegetarian 42135 Vegetarian 36283 Vegan 27356 Maybe vegetarian 16327 Palm oil 14315 May contain palm oil 12543 Maybe vegan 6093
Top 10k most scanned products:
Non-vegan 4629 Vegetarian status unknown 4489 Palm oil free 4277 Palm oil content unknown 3643 Vegan status unknown 2542 Vegetarian 2420 Vegan 1723 Maybe vegetarian 1460 Non-vegetarian 1089 Palm oil 897 May contain palm oil 641 Maybe vegan 564
IT - Italian - Italy
Vegetarian status unknown 3226 Palm oil content unknown 2965 Vegan status unknown 2229 Non-vegan 1492 Palm oil free 748 Vegetarian 507 Vegan 412 Non-vegetarian 295 Palm oil 279 May contain palm oil 208 Maybe vegetarian 172 Maybe vegan 67
Top 10k most scanned products:
Vegetarian status unknown 1320 Palm oil content unknown 1206 Vegan status unknown 932 Non-vegan 578 Palm oil free 290 Vegetarian 205 Vegan 159 Palm oil 118 Non-vegetarian 107 May contain palm oil 81 Maybe vegetarian 63 Maybe vegan 26
NL - Dutch - The Netherlands
Vegetarian status unknown 1663 Palm oil content unknown 1442 Vegan status unknown 1202 Non-vegan 729 Palm oil free 504 Vegetarian 382 Vegan 322 Palm oil 229 Non-vegetarian 161 May contain palm oil 119 Maybe vegetarian 88 Maybe vegan 41
Top 10k most scanned products:
Vegetarian status unknown 447 Palm oil content unknown 364 Vegan status unknown 319 Non-vegan 189 Palm oil free 113 Vegetarian 79 Vegan 67 Palm oil 66 May contain palm oil 42 Non-vegetarian 36 Maybe vegetarian 23 Maybe vegan 10
PT - Portuguese - Portugal
Vegetarian status unknown 1072 Palm oil content unknown 914 Vegan status unknown 768 Non-vegan 454 Palm oil free 187 May contain palm oil 140 Vegetarian 126 Palm oil 105 Vegan 104 Non-vegetarian 97 Maybe vegetarian 51 Maybe vegan 20
Top 10k most scanned products:
Vegetarian status unknown 453 Palm oil content unknown 375 Vegan status unknown 328 Non-vegan 171 Palm oil free 70 May contain palm oil 53 Vegetarian 48 Palm oil 45 Vegan 36 Non-vegetarian 22 Maybe vegetarian 20 Maybe vegan 8