Changes

Jump to navigation Jump to search
6,171 bytes added ,  07:17, 23 October 2023
Line 1: Line 1:  +
[[Category:Taxonomies]]
 
== Introduction ==
 
== Introduction ==
 
== Why? ==  
 
== Why? ==  
Line 128: Line 129:  
* '''Contaminated ingredients''' - a lot of products contain text that are not actual ingredients. Either this is caused by text recognition, which recognizes other elements as well. Or people add text that are not ingredients;
 
* '''Contaminated ingredients''' - a lot of products contain text that are not actual ingredients. Either this is caused by text recognition, which recognizes other elements as well. Or people add text that are not ingredients;
   −
= Maintenance =
+
== Setup ==
In this section the actions and procedures for the maintenance of the taxonomies are described.
+
Adding a new language for ingredients to OFF, starts with the basics: adding a lot of translations. Without this, your language will not be recognised, handles and analysed. There are two ways to add these translations.
 +
=== Adding translations ===
 +
The basic translation work can be done by anyone with a web-browser. You can use the available web-interface for this. For example to translate the dutch ingredients, use this link: [https://world-nl.openfoodfacts.org/ingredients?translate=add]. In this url ''world'' stands for all products in the database and ''nl'' stands for those with dutch as a main language. And the go through the provided list of ingredients. Be careful, as you might easily introduce wrong translations, which are not actually used on ingredient lists.
    +
=== Editing translations ===
 +
If you have experience with Github, you can edit the necessary files as well on your desktop. The ingredients are split over multiple files, which allows for specific handling:
 +
* [https://github.com/openfoodfacts/openfoodfacts-server/blob/master/taxonomies/ingredients.txt Ingredients taxonomy] - this file contains all the normal ingredients;
 +
* [https://github.com/openfoodfacts/openfoodfacts-server/blob/master/taxonomies/additives.txt Additives] - the file with all the additives;
 +
* [https://github.com/openfoodfacts/openfoodfacts-server/blob/master/taxonomies/allergens.txt Allergens] - the allergens required by legislation;
 +
* [https://github.com/openfoodfacts/openfoodfacts-server/blob/master/taxonomies/amino_acids.txt Amino-acids] - the amino acids, required for infant products;
 +
* [https://github.com/openfoodfacts/openfoodfacts-server/blob/master/taxonomies/minerals.txt Minerals] - the minerals required by legislation;
 +
* [https://github.com/openfoodfacts/openfoodfacts-server/blob/master/taxonomies/nucleotides.txt Nucleotides] - the nucleotides required by legislation;
 +
* [https://github.com/openfoodfacts/openfoodfacts-server/blob/master/taxonomies/other_nutritional_substances.txt Other nutritional substances] - also legislation;
 +
* [https://github.com/openfoodfacts/openfoodfacts-server/blob/master/taxonomies/vitamins.txt Vitamins] - legislation issue;
 +
You will noticed that many ingredients are combined with an adjective either in front or after the ingredient. As many adjectives will occur often, it is not necessary to add all variants to the ingredient taxonomy. This will make the taxonomy smaller and makes processing faster. Several different types of adjectives are identified:
 +
* [https://github.com/openfoodfacts/openfoodfacts-server/blob/master/taxonomies/ingredients_processing.txt Processes] - describe what has be done to an ingredient (roasted, rehydrated, etc.);
 +
* [https://github.com/openfoodfacts/openfoodfacts-server/blob/master/taxonomies/labels.txt Labels] - describe the nature (?) of the ingredient, i.e. organic, fair trade, etc;
 +
* [https://github.com/openfoodfacts/openfoodfacts-server/blob/master/taxonomies/additives_classes.txt Additive classes] - each additive can be assigned to one or more additive class, which describes the role of the additive  in the product;
 +
 +
== Maintenance ==
 +
Once the bulk of the ingredients have been added and translated, the maintenance phase begins. You will quickly see that the system identifies words and phrases as ingredients, when it should not do that. Time for finetuning and maintenance.
 +
 +
=== Detection ===
 +
Finetuning and maintenance starts with detection, detection that an ingredient is not or falsely detected.
 +
* Unknowns - start by creating a list of unknown ingredients with the url: https://nl.openfoodfacts.org/ingredients?status=unknown . This will show all products sold in the Netherlands (nl), with unknown ingredients.
 +
 +
You might find unknown ingredients (prefixed with a language code) in other languages if the main language is not the requested language.
 +
 +
=== Finetuning ===
 +
Finetuing allows you to tell the system, where the ingredients start and end, which words should not be taken into account, etc. You might regularly return to this task, as the system analyses more and more ingredient lists.
 +
* Adapt file [https://github.com/openfoodfacts/openfoodfacts-server/blob/master/lib/ProductOpener/Ingredients.pm ingredients.pm] - this file contains various elements, which are used to analyse the ingredients of a product (in the order they appear in the file):
 +
** Traces (''%may_contain_regexps'') - is the start of a phrase that introduces traces;
 +
** Allergens (''%contains_regexps'') - appears after an ingredient, to indicate an allergen;
 +
** Abreviations (''%abbreviations'') - commonly used abbreviations in ingredient lists;
 +
** Or (''%of'') - an optional choice between ingredients, eg '''colza or sunflower oil''';
 +
** And (''%and'') - two ingredients, not separated by a delimiter, eg '''herbs and spices''';
 +
** And of (''%and_of'') -
 +
** And/or (''%and_or'') -
 +
** The (''%the'') - the [https://en.wikipedia.org/wiki/The grammatical article] of the language.
 +
** Ignore after percent (''%ignore_strings_after_percent'') - a phrase that can be discard after it appears a %-sign;
 +
** Ignore phrases (''%ignore_regexps'') - phrases that can be ignored (black list);
 +
** Ingredients start (''%phrases_before_ingredients_list'') - this word is used as the beginning a the ingredient list, and helps OFF determine where the recognition should start.
 +
** Ingredients start all caps (''%phrases_before_ingredients_list_uppercase'') - For example in[https://static.openfoodfacts.org/images/products/002/222/420/0820/ingredients_en.4.full.jpg this ingredient list], the lis starts with '''INGREDIENTS''';
 +
** After ingredients (''%phrases_after_ingredients_list'') - an ingredient list can end in many different ways. These can be listed here;
 +
** Wrong dashes (%prefixes_before_dash) - this allows words separated by dash to be combined, eg '''demi - écrémé''', wil be changed to '''demi-écrémé'''.
 +
You might require special handling, which in turn requires specific code. Please contact the OFF'ers on Slack for that (#ingredients).
 +
 +
=== Adding ===
 
- Finding unknown ingredients
 
- Finding unknown ingredients
 
- Adding processes
 
- Adding processes
Line 136: Line 183:  
- Adding black listed ingredients
 
- Adding black listed ingredients
 
- Language transposes
 
- Language transposes
 +
 +
== Related pages==
 +
There is more documentation that is related ingredients and the corresponding taxonomy:
 +
* [[Ingredients Events list]]
 +
[[Category:Taxonomies]]
 +
[[Category:Ingredients]]
3,393

edits

Navigation menu