Changes

Jump to navigation Jump to search
no edit summary
Line 23: Line 23:  
* Identifying food products that can be or cannot be eaten by people following specific diets:
 
* Identifying food products that can be or cannot be eaten by people following specific diets:
 
** Vegetarian, vegan
 
** Vegetarian, vegan
** Casher, Halal
+
** Kosher, Halal
** Palm oil
+
** No palm oil
 
* Estimating the carbon impact of ingredients
 
* Estimating the carbon impact of ingredients
 
* Translating ingredients lists
 
* Translating ingredients lists
Line 102: Line 102:  
** Only a few adhoc tests run during builds
 
** Only a few adhoc tests run during builds
 
** Test sets needs to be created
 
** Test sets needs to be created
 +
More details can be found on the page [[Ingredients List Cutting]].
    
==== Validation and/or correction by users ====
 
==== Validation and/or correction by users ====
Line 134: Line 135:     
== Ingredients analysis ==
 
== Ingredients analysis ==
 +
 +
=== Ingredients taxonomy ===
 +
 +
Ingredients analysis is about matching the ingredient list to known ingredients in our multilingual ingredients taxonomy.
 +
 +
So a pre-requisite for good ingredients analysis is to have a comprehensive ingredients taxonomy that includes translations and synonyms in the target language.
 +
 +
The taxonomy is on GitHub (it is a very big file): https://github.com/openfoodfacts/openfoodfacts-server/blob/master/taxonomies/ingredients.txt
 +
 +
=== Ingredients processing taxonomy ===
 +
 +
A lot of ingredients list also contain information on how the ingredients have been processed (e.g. "cooked pork meat", "sliced tomatoes", "powdered garlic").
 +
 +
Instead of listing all possible combinations of processing for each ingredient in the ingredients taxonomy, we have created instead a taxonomy of processing methods that we use during ingredients parsing:
 +
 +
https://github.com/openfoodfacts/openfoodfacts-server/blob/master/taxonomies/ingredients_processing.txt
    
=== Steps for ingredients analysis ===
 
=== Steps for ingredients analysis ===
Line 171: Line 188:  
*** lib/ProductOpener/Ingredients.pm - extract_ingredients_from_text()
 
*** lib/ProductOpener/Ingredients.pm - extract_ingredients_from_text()
   −
Result:
+
==== Ingredient percent analysis ====
* Add /api/v0 to get JSON results through API: https://fr.openfoodfacts.org/api/v0/produit/5000112558265/coca-cola-zero
+
* Goal: for each ingredient and sub-ingredient, we compute the minimum and maximum absolute percent
* ingredients:
+
* Constraints:
 +
** Some ingredients have a % specified
 +
** Ingredients are listed in descending order of quantity
 +
* Current solution
 +
** Perl code in lib/ProductOpener/Ingredients.pm - compute_ingredients_percent_values()
 +
** We use a recursive function to go through ingredients, sub-ingredients, sub-sub-ingredients etc.
 +
** For each list of ingredients (or sub-ingredients), we assign starting min, max or specific percent values that comply with the constraints
 +
*** If a % is specified, we use it
 +
*** otherwise the min is set to 0 and the max to the max of the parent ingredient (or 100% if there is no parent)
 +
** We then apply logic rules based on the constraints:
 +
*** The max of an ingredient must be lower or equal to the max of the ingredient that appears before
 +
*** The max of an ingredient must be lower or equal to the  total max minus the sum of the minimums of all ingredients that appear before
 +
*** The max of the 3rd ingredient has a max inferior or equal to the max of the 1st ingredient divided by 2 (and similarly the 4th ingredient has a max inferior or equal to the max of the 1st ingredient divided by 3, etc.)
 +
*** The min of an ingredient must be greater or equal to the total min minus the sum of the maximums of all ingredients that appear before, divided by the number of ingredients that appear after + the current ingredient
 +
*** The min of an ingredient must be greater or equal to the mean of an ingredient that appears after
 +
*** The max of an ingredient must be lower or equal to the total max minus the sum of the minimums of all the ingredients after, divided by the number of ingredients that appear before + the current ingredient
 +
*** The min of the first ingredient in the list must be greater or equal to the total min minus the sum of the maximums of all the ingredients after
 +
** We then reapply all those rules as long as we can apply a new one, based on new min and max values of the ingredients
 +
* Issues:
 +
** We may end up with impossible values if the ingredients list was not analyzed correctly (e.g. wrong nesting, bad % etc.)
 +
*** in that case we delete the min and max values
 +
 
 +
==== Ingredients analysis result: ====
 +
* Add /api/v2 to get JSON results through API: [https://fr.openfoodfacts.org/api/v0/produit/5000112558265/coca-cola-zero https://fr.openfoodfacts.org/api/v2/produit/5000112558265/coca-cola-zero]
 +
*  
   −
<pre>
+
* ingredients: [
ingredients: [
+
** {
{
+
*** id: "en:carbonated-water",
vegetarian: "yes",
+
*** percent_estimate: 56.25,
text: "Eau gazéifiée",
+
*** percent_max: 100,
id: "en:carbonated-water",
+
*** percent_min: 12.5,
rank: 1,
+
*** text: "Eau gazéifiée",
vegan: "yes"
+
*** vegan: "yes",
},
+
*** vegetarian: "yes" },
{
+
** {
rank: 2,
+
*** id: "en:colour",
id: "en:colour",
+
*** ingredients: [
text: "colorant"
+
**** {
},
+
***** id: "en:e150d",
{
+
***** percent_estimate: 21.875,
vegetarian: "yes",
+
***** percent_max: 50,
id: "en:e150d",
+
***** percent_min: 0,
rank: 3,
+
***** text: "e150d",
text: "e150d",
+
***** vegan: "yes",
vegan: "yes"
+
***** vegetarian: "yes" }],
},
+
*** percent_estimate: 21.875,
{
+
*** percent_max: 50,
text: "acidifiants",
+
*** percent_min: 0,
rank: 4,
+
*** text: "colorant" },
id: "en:acid"
+
** {
},
+
*** id: "en:acid",
{
+
*** ingredients: [
vegan: "yes",
+
**** {
vegetarian: "yes",
+
***** id: "en:e338",
rank: 5,
+
***** percent_estimate: 10.9375,
id: "en:e338",
+
***** percent_max: 33.3333333333333,
text: "acide phosphorique"
+
***** percent_min: 0,
},
+
***** text: "acide phosphorique",
{
+
***** vegan: "yes",
text: "citrate de sodium",
+
***** vegetarian: "yes" }],
rank: 6,
+
*** percent_estimate: 10.9375,
id: "en:sodium-citrate"
+
*** percent_max: 33.3333333333333,
},
+
*** percent_min: 0,
{
+
*** text: "acidifiants" },
id: "en:sweetener",
+
** {
rank: 7,
+
*** id: "en:sodium-citrate",
text: "édulcorants"
+
*** percent_estimate: 5.46875,
},
+
*** percent_max: 25,
{
+
*** percent_min: 0,
vegan: "yes",
+
*** text: "citrate de sodium" },
id: "en:e951",
+
** {
rank: 8,
+
*** id: "en:sweetener",
text: "aspartame",
+
*** ingredients: [
vegetarian: "yes"
+
**** {
},
+
***** id: "en:e951",
{
+
***** percent_estimate: 2.734375,
vegetarian: "yes",
+
***** percent_max: 20,
text: "acésulfame-K",
+
***** percent_min: 0,
id: "en:e950",
+
***** text: "aspartame",
rank: 9,
+
***** vegan: "yes",
vegan: "yes"
+
***** vegetarian: "yes" }],
},
+
*** percent_estimate: 2.734375,
{
+
*** percent_max: 20,
vegetarian: "maybe",
+
*** percent_min: 0,
rank: 10,
+
*** text: "édulcorants" },
id: "en:natural-flavouring",
+
** {
text: "arômes naturels",
+
*** id: "en:e950",
vegan: "maybe"
+
*** percent_estimate: 1.3671875,
},
+
*** percent_max: 16.6666666666667,
{
+
*** percent_min: 0,
id: "en:vegetable-extract",
+
*** text: "acésulfame-K",
rank: 11,
+
*** vegan: "yes",
text: "extraits végétaux"
+
*** vegetarian: "yes" },
},
+
** {
{
+
*** id: "en:natural-flavouring",
text: "dont caféine",
+
*** ingredients: [
id: "en:caffeine",
+
**** {
rank: 12,
+
***** id: "en:extract",
vegetarian: "yes",
+
***** labels: "en:vegan",
vegan: "yes"
+
***** percent_estimate: 0.68359375,
}
+
***** percent_max: 14.2857142857143,
],
+
***** percent_min: 0,
</pre>
+
***** text: "extraits",
 +
***** vegan: "en:yes",
 +
***** vegetarian: "en:yes" }],
 +
*** percent_estimate: 0.68359375,
 +
*** percent_max: 14.2857142857143,
 +
*** percent_min: 0,
 +
*** text: "arômes naturels",
 +
*** vegan: "maybe",
 +
*** vegetarian: "maybe" },
 +
** {
 +
*** id: "en:caffeine",
 +
*** percent_estimate: 0.68359375,
 +
*** percent_max: 12.5,
 +
*** percent_min: 0,
 +
*** text: "dont caféine",
 +
*** vegan: "yes",
 +
*** vegetarian: "yes" }],
    
== End to end metrics ==
 
== End to end metrics ==
Line 268: Line 325:  
** Can be given for a subset of products
 
** Can be given for a subset of products
 
*** https://fr.openfoodfacts.org/editor/scamark/ingredients?stats=1 (results from product data imported from Scamark / Leclerc)
 
*** https://fr.openfoodfacts.org/editor/scamark/ingredients?stats=1 (results from product data imported from Scamark / Leclerc)
 +
 +
=== Steps to edit and correct errors in a product ===
 +
 +
* Open the product
 +
** If there is no main front picture then select one from the pictures
 +
** If there are no pictures of anything then delete the product
 +
** If it is not a food product then move it to the relevant database (Beauty, Pet.. see 'Non-Food Products' in [[Quality#How_to_move_these_products.3F|How to move these products]])
 +
** Determine if the language of the ingredients matches the main language
 +
** Look to see if the ingredients look right or if it is totally wrong. For example contains the nutrition information, total junk etc. If it's totally wrong then delete the ingredients.
 +
** If an ingredients label isn't selected then try to find one in the main language and cut it to size. If there is no label in any language then the ingredients aren't reliable so delete them.
 +
** Check the ingredients match the ingredients in the label. If not then extract them. If the ingredients are not extractable or readable then delete them.
 +
** If ingredients match then look for the reason why the ingredient is unknown (mis-spelt etc.)
 +
** If it's an unknown ingredient then add it to the ingredients.txt taxonomy file. If the ingredient exists in English then just add the ingredient for the language you are working in.
 +
** If the ingredient doesn't exist in any language then try to find a parent of the ingredient and add it in that section.
 +
** Check the nutritional information if it exists and correct where necessary
 +
** If the ingredients list doesn't match the main language then change the main language and select the tick box (if you are an administrator) to move all data and images to the main language.
 +
 +
* Depending on the level of editing/cleaning, optionally run the list of ingredients through the [https://world.openfoodfacts.org/cgi/test_ingredients_analysis.pl Ingredients Analysis Test]for the language you are working with.
    
=== Results of further ingredient analysis ===
 
=== Results of further ingredient analysis ===
Line 282: Line 357:     
* Number of products for which we are able to make a NOVA determination
 
* Number of products for which we are able to make a NOVA determination
** A NOVA 3 or 4 ingredients automatically makes the product NOVA 3 or 4
+
** An ingredient marked as a NOVA 3 or NOVA 4 marker automatically makes the product NOVA 3 or 4
 
** NOVA classification not made if too many ingredients are unknown (unless we have a NOVA 4 marker)
 
** NOVA classification not made if too many ingredients are unknown (unless we have a NOVA 4 marker)
 
** --> probably not a good indication of the quality of ingredients recognition
 
** --> probably not a good indication of the quality of ingredients recognition
   −
== Ressources ==
+
== Ingredients analysis quality ==
 +
 
 +
[[Ingredients_Analysis_Quality]] explains how we measure and monitor the quality of the ingredients analysis.
 +
 
 +
== Resources ==
    
=== Data ===
 
=== Data ===
Line 294: Line 373:  
** API
 
** API
 
*** product URL on OFF: https://fr.openfoodfacts.org/produit/3560070223145/pistaches-grillees-carrefour
 
*** product URL on OFF: https://fr.openfoodfacts.org/produit/3560070223145/pistaches-grillees-carrefour
*** add "/api/v0" to get JSON result: https://fr.openfoodfacts.org/api/v0/produit/3560070223145/pistaches-grillees-carrefour
+
*** add "/api/v2" to get JSON result: https://fr.openfoodfacts.org/api/v2/produit/3560070223145/pistaches-grillees-carrefour
 
* Sorted lists with counts of individual ingredients for a subset of product
 
* Sorted lists with counts of individual ingredients for a subset of product
 
** https://fr.openfoodfacts.org/ingredients?stats=1
 
** https://fr.openfoodfacts.org/ingredients?stats=1
 
*** Ingredients that are known (they exist in our taxonomy): https://fr.openfoodfacts.org/editor/scamark/ingredients?status=known
 
*** Ingredients that are known (they exist in our taxonomy): https://fr.openfoodfacts.org/editor/scamark/ingredients?status=known
*** Unknown ingredients: https://fr.openfoodfacts.org/editor/scamark/ingredients?status=known
+
*** Unknown ingredients: https://fr.openfoodfacts.org/editor/scamark/ingredients?status=unknown
 
**** OCR and spelling errors
 
**** OCR and spelling errors
 
**** Parsing errors
 
**** Parsing errors
Line 310: Line 389:  
* Definition: https://github.com/openfoodfacts/openfoodfacts-server/blob/master/taxonomies/ingredients.txt
 
* Definition: https://github.com/openfoodfacts/openfoodfacts-server/blob/master/taxonomies/ingredients.txt
 
* JSON result: http://static.openfoodfacts.org/data/taxonomies/ingredients.json
 
* JSON result: http://static.openfoodfacts.org/data/taxonomies/ingredients.json
 +
 +
== How to improve ingredients analysis ==
 +
 +
This page describes [[How to improve ingredients analysis]] (after the text of the ingredients list has been extracted).
 +
 +
[[Category:Ingredients]]
3,393

edits

Navigation menu