Difference between revisions of "✍️ Ingredients Spellcheck"

Latest revision as of 10:10, 6 September 2024

Key dates and numbers

Why it matters

Most ingredient lists are extracted from packaging images using Optical Character Recognition (OCR) techniques. However, due to factors such as poor image quality, product curvature, or light reflections, the OCR process may incorrectly recognize text characters.

These errors affect the Ingredients Extraction and Analysis process, impacting the overall quality of the database.

Recent advancements in Machine Learning, particularly with the development of Large Language Models, offer new opportunities to address this issue, where previous solutions have yielded limited success.

Picture of a product list of ingredients on its packaging.

What we’ve done recently

What we are currently working on

What’s coming up next

How to get involved

⭐ Guidelines

The impact of the Ingredient-Spellcheck needs to be controlled to avoid altering contributions and/or add new errors to "healthy" ingredient names. The text needs to be kept as unchanged as possible. From the different types of errors observed across products, we came up with these spellcheck guidelines:

Whitespaces between words and percentages shouldn't be corrected.
- Example: "salt 0,2 %"
If a percentage is ambiguous, it shouldn't be corrected.
- Examples: "15 0%" ; "396"
A whitespace involving a percentage shouldn't be modified, except if the digit is stuck in the previous word.
- Example: "cheese1.9%" → "cheese 1.9%"
'_' shouldn't be removed from ingredients that are enclosed in '_' in order to specify allergens.
- Example: "_cacahuetes_" → "_cacahuetes_"
- Example: "_Cacahuetes_ con cáscara tostado. _Trazas de frutos de cáscara_." → "_Cacahuetes_ con cáscara tostado. Trazas de frutos de cáscara."
Some percentages were badly parsed by the OCR. Since we cannot be sure about what is the right value, it is preferable to keep it as it is.
In Romanian, the characters 'ş' (U+015F) and 'ţ' (U+0163) should be retrieved by the Spellcheck when necessary,
Uppercase letters shouldn't be modified in order to add/remove accents.
- Examples: "ECOSSE" ; "ÉCOSSE"
Lowercase letters should be modified in order to add missing accents.
'*' shouldn't be removed.
- Example: "Schweinefleisch*"
Whitespaces shouldn't been modified, except if two words are stuck to each other.
- Example: "rizbrun" → "riz brun"
Letters shouldn't be converted to uppercase/lowercase, except if a word is after a period or if a word is a proper noun.
- Example: "[...] orange.trace de [...]" → "[...] orange. Trace de [...]"
- Example: "france" → "France"
In French, the character "oe" or 'œ' should remain unchanged after correction. If it is missing, it should be replaced by default by 'œ'.
- Examples: "œuf" ; "bœuf"
Commas and other word separators (, -, .) should be added to distinct ingredients.
- Example: "citric acid electrolytes (salt, magnesium and calcium chlorides, mono-potassion phosphate)" → "citric acid, electrolytes (salt, magnesium and calcium chlorides, mono-potassion phosphate)"
The existing punctuation at the end of the list of ingredients shouldn't be modified. In particular, a period shouldn't be added at the end of the list of ingredients.
A ':' should be added if it is missing between the category and name of an additive.
- Example: "conservateur nitrite de sodium" → "conservateur: nitrite de sodium"

@@ Line 2: / Line 2: @@
 [[Category:Ingredients]]
+== Key dates and numbers ==
+== Why it matters ==
 Most ingredient lists are extracted from packaging images using [https://en.wikipedia.org/wiki/Optical_character_recognition Optical Character Recognition] (OCR) techniques. However, due to factors such as poor image quality, product curvature, or light reflections, the OCR process may incorrectly recognize text characters.
@@ Line 10: / Line 12: @@
 [[File:Ingredients picture.png|thumb|Picture of a product list of ingredients on its packaging.]]
+== What we’ve done recently ==
+== What we are currently working on ==
+== What’s coming up next ==
+== How to get involved ==
 == ⭐ Guidelines ==
@@ Line 20: / Line 26: @@
 * A whitespace involving a percentage shouldn't be modified, except if the digit is stuck in the previous word.
 ** Example: "''cheese1.9%''" → "''cheese 1.9%''"
-* Some ingredients are enclosed with `_`, such as `_milk_` or `_cacahuetes_`, to detect allergens. Should remain unchanged. However, in the case it is not an ingredient, such as `_Cacahuetes_ con cáscara tostado. _Trazas de frutos de cáscara_.`, it needs to be modified into `_Cacahuetes_ con cáscara tostado. Trazas de frutos de cáscara.`;
+* '_' shouldn't be removed from ingredients that are enclosed in '_' in order to specify allergens.
+** Example: "''_cacahuetes_''" → "''_cacahuetes_''"
+** Example: "''_Cacahuetes_ con cáscara tostado. _Trazas de frutos de cáscara_.''" → "''_Cacahuetes_ con cáscara tostado. Trazas de frutos de cáscara.''"
 * Some percentages were badly parsed by the OCR. Since we cannot be sure about what is the right value, it is preferable to keep it as it is.
-* Accents and other language specific punctuations:
+* In Romanian, the characters 'ş' (U+015F) and 'ţ' (U+0163) should be retrieved by the Spellcheck when necessary,
-* In Romanian, the characters ["ş" (351), "ţ" (355)] (ASCII id)  should be retrieved by the Spellcheck when necessary,
+* Uppercase letters shouldn't be modified in order to add/remove accents.
-* Uppercase letters shouldn't be modified.
 ** Examples: "''ECOSSE''" ; "''ÉCOSSE''"
-* If lowercase, accent should be added if missing.
+* Lowercase letters should be modified in order to add missing accents.
 * '*' shouldn't be removed.
 ** Example: "''Schweinefleisch*''"
-* Whitespaces shouldn't been modified, except in the following cases:
+* Whitespaces shouldn't been modified, except if two words are stuck to each other.
-** when two words are stuck to each other.
+** Example: "''rizbrun''" → "''riz brun''"
-*** Example: "''rizbrun''" → "''riz brun''"
+* Letters shouldn't be converted to uppercase/lowercase, except if a word is after a period or if a word is a proper noun.
-* Regarding uppercases and lowercases, since the spellcheck should modify at least as possible lists of ingredient, we don't modify uppercases or lowercases except for the following reasons:
+** Example: "''[...] orange.trace de [...]''" → "''[...] orange. Trace de [...]''"
-** after a period: "orange.trace de..." → "orange. Trace de..."
+** Example: "''france''" → "''France''"
-** if it's a proper noun: "france" → "France"
 * In French, the character "oe" or 'œ' should remain unchanged after correction. If it is missing, it should be replaced by default by 'œ'.
 ** Examples: "''œuf''" ; "''bœuf''"
 * Commas and other word separators (, -, .) should be added to distinct ingredients.
 ** Example: "''citric acid electrolytes (salt, magnesium and calcium chlorides, mono-potassion phosphate)''" → "''citric acid, electrolytes (salt, magnesium and calcium chlorides, mono-potassion phosphate)''"
-* We don't add a period or modify the existing punctuation at the end of the list of ingredients.
+* The existing punctuation at the end of the list of ingredients shouldn't be modified. In particular, a period shouldn't be added at the end of the list of ingredients.
-* A ':' should be added in the following cases:
+* A ':' should be added if it is missing between the category and name of an additive.
-** it is missing between the additive category and the additive name
+** Example: "''conservateur nitrite de sodium''" → "''conservateur: nitrite de sodium''"
-*** Example: "''conservateur nitrite de sodium''" → "''conservateur: nitrite de sodium''"