Changes

Jump to navigation Jump to search
3,559 bytes added ,  07:17, 23 October 2023
Line 1: Line 1:  +
[[Category:Taxonomies]]
 
== Introduction ==
 
== Introduction ==
 
== Why? ==  
 
== Why? ==  
Line 127: Line 128:  
* '''Translation bootstrapping''' - in order to get up to speed with the translations, we use the Wikipedia and Wikidata to find translations. This can result in the wrong translations as it can be something different than what is found on the package. So be careful for wikipedia disambiguation strings, latin species names, etc. In the long run these should be superseded by what is fond on the products.
 
* '''Translation bootstrapping''' - in order to get up to speed with the translations, we use the Wikipedia and Wikidata to find translations. This can result in the wrong translations as it can be something different than what is found on the package. So be careful for wikipedia disambiguation strings, latin species names, etc. In the long run these should be superseded by what is fond on the products.
 
* '''Contaminated ingredients''' - a lot of products contain text that are not actual ingredients. Either this is caused by text recognition, which recognizes other elements as well. Or people add text that are not ingredients;
 
* '''Contaminated ingredients''' - a lot of products contain text that are not actual ingredients. Either this is caused by text recognition, which recognizes other elements as well. Or people add text that are not ingredients;
 +
 +
== Setup ==
 +
Adding a new language for ingredients to OFF, starts with the basics: adding a lot of translations. Without this, your language will not be recognised, handles and analysed. There are two ways to add these translations.
 +
=== Adding translations ===
 +
The basic translation work can be done by anyone with a web-browser. You can use the available web-interface for this. For example to translate the dutch ingredients, use this link: [https://world-nl.openfoodfacts.org/ingredients?translate=add]. In this url ''world'' stands for all products in the database and ''nl'' stands for those with dutch as a main language. And the go through the provided list of ingredients. Be careful, as you might easily introduce wrong translations, which are not actually used on ingredient lists.
 +
 +
=== Editing translations ===
 +
If you have experience with Github, you can edit the necessary files as well on your desktop. The ingredients are split over multiple files, which allows for specific handling:
 +
* [https://github.com/openfoodfacts/openfoodfacts-server/blob/master/taxonomies/ingredients.txt Ingredients taxonomy] - this file contains all the normal ingredients;
 +
* [https://github.com/openfoodfacts/openfoodfacts-server/blob/master/taxonomies/additives.txt Additives] - the file with all the additives;
 +
* [https://github.com/openfoodfacts/openfoodfacts-server/blob/master/taxonomies/allergens.txt Allergens] - the allergens required by legislation;
 +
* [https://github.com/openfoodfacts/openfoodfacts-server/blob/master/taxonomies/amino_acids.txt Amino-acids] - the amino acids, required for infant products;
 +
* [https://github.com/openfoodfacts/openfoodfacts-server/blob/master/taxonomies/minerals.txt Minerals] - the minerals required by legislation;
 +
* [https://github.com/openfoodfacts/openfoodfacts-server/blob/master/taxonomies/nucleotides.txt Nucleotides] - the nucleotides required by legislation;
 +
* [https://github.com/openfoodfacts/openfoodfacts-server/blob/master/taxonomies/other_nutritional_substances.txt Other nutritional substances] - also legislation;
 +
* [https://github.com/openfoodfacts/openfoodfacts-server/blob/master/taxonomies/vitamins.txt Vitamins] - legislation issue;
 +
You will noticed that many ingredients are combined with an adjective either in front or after the ingredient. As many adjectives will occur often, it is not necessary to add all variants to the ingredient taxonomy. This will make the taxonomy smaller and makes processing faster. Several different types of adjectives are identified:
 +
* [https://github.com/openfoodfacts/openfoodfacts-server/blob/master/taxonomies/ingredients_processing.txt Processes] - describe what has be done to an ingredient (roasted, rehydrated, etc.);
 +
* [https://github.com/openfoodfacts/openfoodfacts-server/blob/master/taxonomies/labels.txt Labels] - describe the nature (?) of the ingredient, i.e. organic, fair trade, etc;
 +
* [https://github.com/openfoodfacts/openfoodfacts-server/blob/master/taxonomies/additives_classes.txt Additive classes] - each additive can be assigned to one or more additive class, which describes the role of the additive  in the product;
    
== Maintenance ==
 
== Maintenance ==
In this section the actions and procedures for the maintenance of the taxonomies are described.
+
Once the bulk of the ingredients have been added and translated, the maintenance phase begins. You will quickly see that the system identifies words and phrases as ingredients, when it should not do that. Time for finetuning and maintenance.
   −
=== Bootstrap ===
+
=== Detection ===
Before starting with the maintenance action, it is first necessary to get the ingredients your language going. The next steps can be taken:
+
Finetuning and maintenance starts with detection, detection that an ingredient is not or falsely detected.
 +
* Unknowns - start by creating a list of unknown ingredients with the url: https://nl.openfoodfacts.org/ingredients?status=unknown . This will show all products sold in the Netherlands (nl), with unknown ingredients.
 +
 
 +
You might find unknown ingredients (prefixed with a language code) in other languages if the main language is not the requested language.
 +
 
 +
=== Finetuning ===
 +
Finetuing allows you to tell the system, where the ingredients start and end, which words should not be taken into account, etc. You might regularly return to this task, as the system analyses more and more ingredient lists.
 
* Adapt file [https://github.com/openfoodfacts/openfoodfacts-server/blob/master/lib/ProductOpener/Ingredients.pm ingredients.pm] - this file contains various elements, which are used to analyse the ingredients of a product (in the order they appear in the file):
 
* Adapt file [https://github.com/openfoodfacts/openfoodfacts-server/blob/master/lib/ProductOpener/Ingredients.pm ingredients.pm] - this file contains various elements, which are used to analyse the ingredients of a product (in the order they appear in the file):
 
** Traces (''%may_contain_regexps'') - is the start of a phrase that introduces traces;
 
** Traces (''%may_contain_regexps'') - is the start of a phrase that introduces traces;
Line 149: Line 176:  
** Wrong dashes (%prefixes_before_dash) - this allows words separated by dash to be combined, eg '''demi - écrémé''', wil be changed to '''demi-écrémé'''.
 
** Wrong dashes (%prefixes_before_dash) - this allows words separated by dash to be combined, eg '''demi - écrémé''', wil be changed to '''demi-écrémé'''.
 
You might require special handling, which in turn requires specific code. Please contact the OFF'ers on Slack for that (#ingredients).
 
You might require special handling, which in turn requires specific code. Please contact the OFF'ers on Slack for that (#ingredients).
* Systematically add the translations - you can use the available web-interface for this. For example to translate the dutch ingredients, use this link: [https://world-nl.openfoodfacts.org/ingredients?translate=add]. In this url ''world'' stands for all products in the database and ''nl'' stands for those with dutch as a main language. And the go through the provided list of ingredients. Be careful, as you might easily introduce wrong translations, which are not actually used on ingredient lists.
     −
=== Actions ===
+
=== Adding ===
 
- Finding unknown ingredients
 
- Finding unknown ingredients
 
- Adding processes
 
- Adding processes
Line 157: Line 183:  
- Adding black listed ingredients
 
- Adding black listed ingredients
 
- Language transposes
 
- Language transposes
 +
 +
== Related pages==
 +
There is more documentation that is related ingredients and the corresponding taxonomy:
 +
* [[Ingredients Events list]]
 +
[[Category:Taxonomies]]
 +
[[Category:Ingredients]]
3,393

edits

Navigation menu