From Open Food Facts wiki
Jump to navigation
Jump to search
Ingredients parsing and recognition
For each product, the ingredients list is parsed to separate each ingredient. Ingredients that we can match to our multilingual ingredients taxonomy as marked as "known", others as "unknown".
There are different reasons an ingredient can be marked as unknown:
- The input ingredient list is incorrect (mispellings etc.)
- The ingredients list contains other things than ingredients and we have not been able to split it at the right place.
- We have not been able to parse a particular sentence structure or formating (e.g. "a drop of delicious honey with a shower of powder sugar")
- The ingredient is not yet present in our taxonomy (or not with the right synonym or in the right language)
In the table below,
- the first column of numbers corresponds to the number of unique ingredients accross all products.
- And the second column of number corresponds to the number of occurrences of those ingredients. e.g. if a specific ingredient appears in 5 products, it is counted 5 times.
Retrieved on 2020-03-05:
EN - English - UK
|
2020-03-05 |
2020-05-14 |
2020-10-31
|
All products: |
Type Unique tags Occurrences
known 1702 (8.77%) 104818 (79.85%)
unknown 17706 (91.23%) 26448 (20.15%)
all 19409 (100.00%) 131266 (100.00%)
|
Type Unique tags Occurrences
known 1787 (8.17%) 124626 (80.82%)
unknown 20099 (91.83%) 29578 (19.18%)
all 21886 (100.00%) 154204 (100.00%)
|
Type Unique tags Occurrences
known 2262 (14.84%) 197422 (92.27%)
unknown 12977 (85.16%) 16548 (7.73%)
all 15239 (100.00%) 213970 (100.00%)
|
Top 10k most scanned products: |
known 1012 (20.67%) 19061 (79.38%)
unknown 3884 (79.31%) 4952 (20.62%)
all 4897 (100.00%) 24013 (100.00%)
|
known 1055 (22.09%) 21216 (81.93%)
unknown 3720 (77.91%) 4678 (18.07%)
all 4775 (100.00%) 25894 (100.00%)
|
known 1347 (44.87%) 28948 (93.92%)
unknown 1655 (55.13%) 1874 (6.08%)
all 3002 (100.00%) 30822 (100.00%)
|
EN - English - US
Note: ~300K US products were imported from the USDA in April, so results may not be very comparable.
|
2020-03-21 |
2020-05-14 |
2020-10-31
|
All products: |
Type Unique tags Occurrences
known 2061 (2.12%) 2677857 (82.12%)
unknown 95216 (97.88%) 583085 (17.88%)
all 97278 (100.00%) 3260942 (100.00%)
|
Type Unique tags Occurrences
known 2218 (1.79%) 6457065 (84.67%)
unknown 122007 (98.21%) 1168962 (15.33%)
all 124225 (100.00%) 7626027 (100.00%)
|
Type Unique tags Occurrences
known 2787 (2.11%) 7435946 (90.52%)
unknown 129044 (97.89%) 778751 (9.48%)
all 131831 (100.00%) 8214697 (100.00%)
|
Top 10k most scanned products: |
known 983 (18.16%) 25484 (77.58%)
unknown 4430 (81.82%) 7364 (22.42%)
all 5414 (100.00%) 32848 (100.00%)
|
known 1055 (20.52%) 38068 (82.02%)
unknown 4086 (79.48%) 8347 (17.98%)
all 5141 (100.00%) 46415 (100.00%)
|
known 1275 (24.46%) 43918 (87.18%)
unknown 3938 (75.54%) 6456 (12.82%)
all 5213 (100.00%) 50374 (100.00%)
|
DE - German - Germany
|
2020-03-05 |
2020-05-14 |
2020-10-31
|
All products: |
Type Unique tags Occurrences
known 2139 (5.36%) 363967 (84.75%)
unknown 37790 (94.64%) 65503 (15.25%)
all 39930 (100.00%) 429470 (100.00%)
|
Type Unique tags Occurrences
known 2233 (5.62%) 417604 (86.72%)
unknown 37522 (94.38%) 63935 (13.28%)
all 39755 (100.00%) 481539 (100.00%)
|
Type Unique tags Occurrences
known 2491 (5.97%) 518324 (89.03%)
unknown 39239 (94.03%) 63880 (10.97%)
all 41730 (100.00%) 582204 (100.00%)
|
Top 10k most scanned products: |
known 1435 (14.73%) 69066 (85.53%)
unknown 8309 (85.26%) 11689 (14.47%)
all 9745 (100.00%) 80755 (100.00%)
|
known 1511 (16.88%) 75653 (88.13%)
unknown 7439 (83.12%) 10187 (11.87%)
all 8950 (100.00%) 85840 (100.00%)
|
known 1624 (19.65%) 85314 (90.61%)
unknown 6640 (80.35%) 8845 (9.39%)
all 8264 (100.00%) 94159 (100.00%)
|
ES - Spanish - Spain
|
2020-03-05 |
2020-05-14 |
2020-10-31
|
All products: |
Type Unique tags Occurrences
known 1970 (4.25%) 176101 (71.95%)
unknown 44352 (95.75%) 68659 (28.05%)
all 46323 (100.00%) 244760 (100.00%)
|
Type Unique tags Occurrences
known 2059 (4.29%) 230207 (76.47%)
unknown 45977 (95.71%) 70846 (23.53%)
all 48036 (100.00%) 301053 (100.00%)
|
Type Unique tags Occurrences
known 2494 (9.14%) 314030 (90.21%)
unknown 24786 (90.86%) 34086 (9.79%)
all 27280 (100.00%) 348116 (100.00%)
|
Top 10k most scanned products: |
known 1262 (10.47%) 48797 (75.99%)
unknown 10793 (89.52%) 15417 (24.01%)
all 12056 (100.00%) 64214 (100.00%)
|
known 1354 (12.19%) 55402 (80.84%)
unknown 9751 (87.81%) 13134 (19.16%)
all 11105 (100.00%) 68536 (100.00%)
|
known 1609 (25.94%) 72353 (92.96%)
unknown 4593 (74.06%) 5478 (7.04%)
all 6202 (100.00%) 77831 (100.00%)
|
FI - Finnish - FI
|
2020-03-05 |
2020-05-14 |
2020-10-31
|
All products: |
Type Unique tags Occurrences
known 829 (43.96%) 11069 (89.15%)
unknown 1056 (55.99%) 1347 (10.85%)
all 1886 (100.00%) 12416 (100.00%)
|
Type Unique tags Occurrences
known 910 (45.09%) 15650 (91.43%)
unknown 1108 (54.91%) 1466 (8.57%)
all 2018 (100.00%) 17116 (100.00%)
|
Type Unique tags Occurrences
known 1285 (56.56%) 25597 (95.22%)
unknown 987 (43.44%) 1284 (4.78%)
all 2272 (100.00%) 26881 (100.00%)
|
Top 10k most scanned products:
|
known 566 (61.66%) 3465 (89.07%)
unknown 351 (38.24%) 425 (10.93%)
all 918 (100.00%) 3890 (100.00%)
|
known 581 (67.40%) 3694 (91.96%)
unknown 281 (32.60%) 323 (8.04%)
all 862 (100.00%) 4017 (100.00%)
|
known 704 (85.02%) 4276 (96.79%)
unknown 124 (14.98%) 142 (3.21%)
all 828 (100.00%) 4418 (100.00%)
|
FR - French - France
|
2020-03-05 |
2020-05-14 |
2020-10-31
|
All products: |
Type Unique tags Occurrences
known 3825 (0.85%) 3708820 (83.40%)
unknown 446838 (99.15%) 738291 (16.60%)
all 450664 (100.00%) 4447111 (100.00%)
|
Type Unique tags Occurrences
known 3793 (0.92%) 3832574 (85.55%)
unknown 406901 (99.08%) 647239 (14.45%)
all 410694 (100.00%) 4479813 (100.00%)
|
Type Unique tags Occurrences
known 4149 (1.08%) 4134163 (87.27%)
unknown 380115 (98.92%) 603291 (12.73%)
all 384264 (100.00%) 4737454 (100.00%)
|
Top 10k most scanned products: |
known 2208 (22.34%) 184218 (94.52%)
unknown 7673 (77.65%) 10686 (5.48%)
all 9882 (100.00%) 194904 (100.00%)
|
known 2260 (24.29%) 186642 (95.08%)
unknown 7044 (75.71%) 9655 (4.92%)
all 9304 (100.00%) 196297 (100.00%)
|
known 2360 (26.98%) 193659 (95.71%)
unknown 6387 (73.02%) 8684 (4.29%)
all 8747 (100.00%) 202343 (100.00%)
|
IT - Italian - Italy
|
2020-03-05 |
2020-05-14 |
2020-10-31
|
All products: |
Type Unique tags Occurrences
known 1319 (9.43%) 53345 (74.58%)
unknown 12669 (90.56%) 18186 (25.42%)
all 13989 (100.00%) 71531 (100.00%)
|
Type Unique tags Occurrences
known 1418 (9.21%) 67900 (78.54%)
unknown 13976 (90.79%) 18557 (21.46%)
all 15394 (100.00%) 86457 (100.00%)
|
Type Unique tags Occurrences
known 1568 (8.56%) 86863 (78.75%)
unknown 16751 (91.44%) 23446 (21.25%)
all 18319 (100.00%) 110309 (100.00%)
|
Top 10k most scanned products: |
known 911 (13.11%) 20838 (72.21%)
unknown 6039 (86.88%) 8018 (27.79%)
all 6951 (100.00%) 28856 (100.00%)
|
known 964 (14.90%) 24072 (78.11%)
unknown 5507 (85.10%) 6747 (21.89%)
all 6471 (100.00%) 30819 (100.00%)
|
known 1024 (15.27%) 27083 (78.92%)
unknown 5683 (84.73%) 7233 (21.08%)
all 6707 (100.00%) 34316 (100.00%)
|
NL - Dutch - The Netherlands
|
2020-03-05 |
2020-05-14 |
2020-10-31
|
All products: |
Type Unique tags Occurrences
known 1304 (17.03%) 31491 (78.84%)
unknown 6353 (82.96%) 8453 (21.16%)
all 7658 (100.00%) 39944 (100.00%)
|
Type Unique tags Occurrences
known 1391 (17.72%) 36019 (80.81%)
unknown 6461 (82.28%) 8555 (19.19%)
all 7852 (100.00%) 44574 (100.00%)
|
|
Top 10k most scanned products: |
known 755 (24.25%) 8069 (73.71%)
unknown 2358 (75.72%) 2878 (26.29%)
all 3114 (100.00%) 10947 (100.00%)
|
known 797 (29.00%) 9104 (79.28%)
unknown 1951 (71.00%) 2379 (20.72%)
all 2748 (100.00%) 11483 (100.00%)
|
|
PT - Portuguese - Portugal
|
2020-03-05 |
2020-05-14 |
2020-10-31
|
All products: |
Type Unique tags Occurrences
known 918 (16.05%) 15816 (70.50%)
unknown 4800 (83.93%) 6617 (29.50%)
all 5719 (100.00%) 22433 (100.00%)
|
Type Unique tags Occurrences
known 948 (16.10%) 17005 (71.53%)
unknown 4940 (83.90%) 6769 (28.47%)
all 5888 (100.00%) 23774 (100.00%)
|
|
Top 10k most scanned products: |
known 578 (19.37%) 6261 (68.40%)
unknown 2405 (80.60%) 2892 (31.60%)
all 2984 (100.00%) 9153 (100.00%)
|
known 602 (20.40%) 6608 (70.43%)
unknown 2349 (79.60%) 2775 (29.57%)
all 2951 (100.00%) 9383 (100.00%)
|
|
Ingredients analysis
For each product, once we have parsed the ingredients list to separate each ingredient and match them to our taxonomy, we can analyze if the products are vegan, vegetarian, without palm oil etc.
To mark a product as non-vegan etc. we only need to find one non-vegan ingredient. But to mark a product as vegan, we need to be sure that we have been able to correctly recognize all ingredients so that we can verify that all of them are vegan.
Retrieved on 2020-03-05:
EN - English - UK
|
2020-03-05 |
2020-05-14
|
All products: |
Vegetarian status unknown 4814
Palm oil content unknown 4138
Vegan status unknown 3281
Non-vegan 2525
Palm oil free 1286
Vegetarian 903
Palm oil 875
Vegan 725
Non-vegetarian 719
May contain palm oil 314
Maybe vegetarian 177
Maybe vegan 82
|
Vegetarian status unknown 4533
Palm oil content unknown 4513
Vegan status unknown 3251
Non-vegan 2919
Vegetarian 2785
Vegan 2043
Palm oil free 1587
Palm oil 1049
Non-vegetarian 853
May contain palm oil 380
Maybe vegetarian 214
Maybe vegan 102
|
Top 10k most scanned products:
|
Vegetarian status unknown 1016
Palm oil content unknown 817
Vegan status unknown 711
Non-vegan 439
Palm oil free 222
Palm oil 192
Vegetarian 171
Vegan 139
Non-vegetarian 81
May contain palm oil 73
Maybe vegetarian 36
Maybe vegan 15
|
Vegetarian status unknown 835
Palm oil content unknown 816
Vegan status unknown 616
Non-vegan 485
Vegetarian 470
Vegan 314
Palm oil free 264
Palm oil 219
Non-vegetarian 92
May contain palm oil 80
Maybe vegetarian 45
Maybe vegan 18
|
DE - German - Germany
|
2020-03-05 |
2020-05-14
|
All products: |
Vegetarian status unknown 15605
Palm oil content unknown 14599
Vegan status unknown 10377
Non-vegan 10151
Palm oil free 6199
Vegetarian 4394
Vegan 3367
Non-vegetarian 2805
Palm oil 2357
Maybe vegetarian 1486
May contain palm oil 1135
Maybe vegan 395
|
Palm oil content unknown 15370
Vegetarian status unknown 14019
Non-vegan 11373
Vegetarian 9087
Vegan status unknown 8791
Palm oil free 7765
Vegan 7377
Non-vegetarian 3225
Palm oil 2592
Maybe vegetarian 1731
May contain palm oil 1365
Maybe vegan 462
|
Top 10k most scanned products: |
Vegetarian status unknown 2925
Palm oil content unknown 2449
Non-vegan 1903
Vegan status unknown 1874
Palm oil free 1255
Vegetarian 849
Vegan 634
Palm oil 567
Maybe vegetarian 381
Non-vegetarian 362
May contain palm oil 246
Maybe vegan 106
|
Palm oil content unknown 2386
Vegetarian status unknown 2384
Non-vegan 2030
Vegetarian 1626
Palm oil free 1471
Vegan status unknown 1419
Vegan 1249
Palm oil 600
Maybe vegetarian 420
Non-vegetarian 397
May contain palm oil 279
Maybe vegan 118
|
ES - Spanish - Spain
|
2020-03-05 |
2020-05-14
|
All products: |
Vegetarian status unknown 12076
Palm oil content unknown 11192
Vegan status unknown 9812
Non-vegan 3877
Palm oil free 2707
Vegetarian 1906
Vegan 1689
Non-vegetarian 1145
May contain palm oil 978
Palm oil 809
Maybe vegetarian 559
Maybe vegan 308
|
Palm oil content unknown 12221
Vegetarian status unknown 10298
Vegetarian 8136
Vegan 7605
Vegan status unknown 7515
Non-vegan 5481
Palm oil free 4988
Non-vegetarian 1805
May contain palm oil 1171
Palm oil 1022
Maybe vegetarian 664
Maybe vegan 286
|
Top 10k most scanned products: |
Vegetarian status unknown 2733
Palm oil content unknown 2387
Vegan status unknown 2081
Non-vegan 1105
Palm oil free 733
Vegetarian 484
Vegan 394
May contain palm oil 308
Non-vegetarian 272
Palm oil 267
Maybe vegetarian 206
Maybe vegan 115
|
Palm oil content unknown 2235
Vegetarian status unknown 2216
Vegan status unknown 1540
Non-vegan 1316
Vegetarian 1096
Palm oil free 997
Vegan 925
Non-vegetarian 362
May contain palm oil 328
Palm oil 286
Maybe vegetarian 209
Maybe vegan 100
|
FI - Finnish - FI
|
2020-03-05 |
2020-05-14
|
All products: |
Vegetarian status unknown 375
Palm oil content unknown 312
Non-vegan 278
Vegan status unknown 245
Palm oil free 191
Maybe vegetarian 94
Non-vegetarian 91
Palm oil 85
Vegetarian 83
Vegan 76
May contain palm oil 55
Maybe vegan 44
|
Vegetarian status unknown 433
Palm oil content unknown 396
Non-vegan 364
Vegan status unknown 283
Palm oil free 256
Vegetarian 178
Vegan 153
Non-vegetarian 128
Maybe vegetarian 115
Palm oil 112
May contain palm oil 78
Maybe vegan 52
|
Top 10k most scanned products: |
Vegetarian status unknown 101
Palm oil content unknown 93
Non-vegan 77
Vegan status unknown 70
Palm oil free 59
Non-vegetarian 31
Vegetarian 27
Maybe vegetarian 25
Vegan 24
Palm oil 19
Maybe vegan 13
May contain palm oil 13
|
Vegetarian status unknown 88
Palm oil content unknown 86
Non-vegan 79
Palm oil free 72
Vegan status unknown 58
Vegetarian 46
Vegan 41
Non-vegetarian 31
Maybe vegetarian 28
Palm oil 22
Maybe vegan 15
May contain palm oil 13
|
FR - French - France
|
2020-03-05 |
2020-05-14
|
All products: |
Palm oil content unknown 128788
Vegetarian status unknown 127223
Non-vegan 105765
Vegan status unknown 82754
Palm oil free 66322
Non-vegetarian 42135
Vegetarian 36283
Vegan 27356
Maybe vegetarian 16327
Palm oil 14315
May contain palm oil 12543
Maybe vegan 6093
|
Palm oil content unknown 120790
Vegetarian status unknown 118197
Non-vegan 107363
Palm oil free 77233
Vegan status unknown 75502
Vegetarian 50863
Non-vegetarian 42829
Vegan 39716
Maybe vegetarian 17543
Palm oil 14895
May contain palm oil 12870
Maybe vegan 6712
|
Top 10k most scanned products: |
Non-vegan 4629
Vegetarian status unknown 4489
Palm oil free 4277
Palm oil content unknown 3643
Vegan status unknown 2542
Vegetarian 2420
Vegan 1723
Maybe vegetarian 1460
Non-vegetarian 1089
Palm oil 897
May contain palm oil 641
Maybe vegan 564
|
Palm oil free 4895
Non-vegan 4607
Vegetarian status unknown 4029
Palm oil content unknown 3089
Vegetarian 2827
Vegan status unknown 2302
Vegan 1944
Maybe vegetarian 1523
Non-vegetarian 1082
Palm oil 889
Maybe vegan 605
May contain palm oil 594
|
IT - Italian - Italy
|
2020-03-05 |
2020-05-14
|
All products: |
Vegetarian status unknown 3226
Palm oil content unknown 2965
Vegan status unknown 2229
Non-vegan 1492
Palm oil free 748
Vegetarian 507
Vegan 412
Non-vegetarian 295
Palm oil 279
May contain palm oil 208
Maybe vegetarian 172
Maybe vegan 67
|
Vegetarian status unknown 3357
Palm oil content unknown 3055
Vegan status unknown 2267
Non-vegan 1820
Palm oil free 1498
Vegetarian 951
Vegan 770
Non-vegetarian 382
Palm oil 316
Maybe vegetarian 272
May contain palm oil 252
Maybe vegan 102
|
Top 10k most scanned products: |
Vegetarian status unknown 1320
Palm oil content unknown 1206
Vegan status unknown 932
Non-vegan 578
Palm oil free 290
Vegetarian 205
Vegan 159
Palm oil 118
Non-vegetarian 107
May contain palm oil 81
Maybe vegetarian 63
Maybe vegan 26
|
Vegetarian status unknown 1230
Palm oil content unknown 1063
Vegan status unknown 839
Non-vegan 637
Palm oil free 572
Vegetarian 343
Vegan 268
Palm oil 123
Non-vegetarian 120
Maybe vegetarian 91
May contain palm oil 89
Maybe vegan 37
|
NL - Dutch - The Netherlands
|
2020-03-05 |
2020-05-14
|
All products: |
Vegetarian status unknown 1663
Palm oil content unknown 1442
Vegan status unknown 1202
Non-vegan 729
Palm oil free 504
Vegetarian 382
Vegan 322
Palm oil 229
Non-vegetarian 161
May contain palm oil 119
Maybe vegetarian 88
Maybe vegan 41
|
Palm oil content unknown 1506
Vegetarian status unknown 1477
Vegetarian 1066
Vegan status unknown 1027
Vegan 921
Non-vegan 826
Palm oil free 634
Palm oil 251
Non-vegetarian 187
May contain palm oil 129
Maybe vegetarian 91
Maybe vegan 44
|
Top 10k most scanned products: |
Vegetarian status unknown 447
Palm oil content unknown 364
Vegan status unknown 319
Non-vegan 189
Palm oil free 113
Vegetarian 79
Vegan 67
Palm oil 66
May contain palm oil 42
Non-vegetarian 36
Maybe vegetarian 23
Maybe vegan 10
|
Vegetarian status unknown 402
Palm oil content unknown 366
Vegan status unknown 274
Non-vegan 209
Vegetarian 160
Palm oil free 139
Vegan 133
Palm oil 70
Non-vegetarian 41
May contain palm oil 39
Maybe vegetarian 26
Maybe vegan 11
|
PT - Portuguese - Portugal
|
2020-03-05 |
2020-05-14
|
All products:
|
Vegetarian status unknown 1072
Palm oil content unknown 914
Vegan status unknown 768
Non-vegan 454
Palm oil free 187
May contain palm oil 140
Vegetarian 126
Palm oil 105
Vegan 104
Non-vegetarian 97
Maybe vegetarian 51
Maybe vegan 20
|
Vegetarian status unknown 1000
Palm oil content unknown 909
Vegan status unknown 700
Non-vegan 478
Vegetarian 265
Palm oil free 245
Vegan 228
May contain palm oil 138
Palm oil 112
Non-vegetarian 108
Maybe vegetarian 56
Maybe vegan 23
|
Top 10k most scanned products: |
Vegetarian status unknown 453
Palm oil content unknown 375
Vegan status unknown 328
Non-vegan 171
Palm oil free 70
May contain palm oil 53
Vegetarian 48
Palm oil 45
Vegan 36
Non-vegetarian 22
Maybe vegetarian 20
Maybe vegan 8
|
Vegetarian status unknown 411
Palm oil content unknown 357
Vegan status unknown 288
Non-vegan 177
Vegetarian 104
Palm oil free 95
Vegan 85
May contain palm oil 49
Palm oil 48
Maybe vegetarian 24
Non-vegetarian 22
Maybe vegan 11
|
Evolution of the metrics between early March 2020 and mid May 2020
Comparison of key metrics
The comparison table contains the metrics for all products of a country.
Metrics:
- Ingredients parsing and recognition: % of known ingredients for all ingredients occurences.
- Ingredients analysis: known vegetarian status
- = 1 - "Vegetarian status unknown" / ("Vegetarian status unknown" + "Vegetarian" + "Non vegetarian" + "Maybe vegetarian")
Country |
% known ingredients March 2020 |
% known ingredients May 2020 |
% known vegetarian status March 2020 |
% known vegetarian status May 2020
|
UK |
79.75% |
80.82% |
27.20% |
45.94%
|
US |
82.12% |
84.67% |
not recorded |
not recorded
|
DE |
84.75% |
86.72% |
35.76% |
50.04%
|
ES |
71.95% |
76.47% |
23.01% |
50.73%
|
FI |
89.15% |
91.43% |
41.68% |
49.30%
|
FR |
83.40% |
85.55% |
42.68% |
48.47%
|
IT |
74.58% |
78.54% |
23.19% |
32.35%
|
NL |
78.84% |
80.81% |
27.51% |
47.64%
|
PT |
70.50% |
71.53% |
20.36% |
26.74%
|
Observations
Significant progress has been made on all tested countries.
For ingredients parsing and recognition, it becomes harder and harder to get big increases in the % of known ingredients. It is expected as we have focused first on the improvements that had the biggest impact. But even a "small" increase from 83.40% to 85.55% for known ingredients in France corresponds to a drop of the unknown ingredients from 16.60% to 14.65% which is a 11.7% relative drop.
It is very interesting to see that the improvements in ingredients parsing and recognition translate in much more significant ingredients analysis (knowing the vegetarian status of a product) improvements. All countries had a 5% to 25% absolute increase of products with known vegetarian status.