Line 51: |
Line 51: |
| * remove json with error messages | | * remove json with error messages |
| | | |
− | === Process === | + | === How to process images to generate JSONs === |
| <pre> | | <pre> |
| sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[0-9].jpg*"); do sudo python /home/xxxxxxxx.py $f;done' | | sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[0-9].jpg*"); do sudo python /home/xxxxxxxx.py $f;done' |
Line 59: |
Line 59: |
| </pre> | | </pre> |
| | | |
− | === Results === | + | === How to check the results === |
| ==== Number of images ==== | | ==== Number of images ==== |
− | 46715+128832+59961+180420 = 415928(edited) | + | 46715+128832+59961+180420 = 415928 (edited) |
| <pre> | | <pre> |
| teolemon@ks3095298:/home/off/html/images/products$ ls -lR /home/off/html/images/products/ | find . -name 'nutrition.*.full.jpg' -type f | wc -l | | teolemon@ks3095298:/home/off/html/images/products$ ls -lR /home/off/html/images/products/ | find . -name 'nutrition.*.full.jpg' -type f | wc -l |
Line 78: |
Line 78: |
| </pre> | | </pre> |
| | | |
− | ==== Number of files ==== | + | ==== Number of files in the directory ==== |
| | | |
| <pre> | | <pre> |
Line 94: |
Line 94: |
| | | |
| </pre> | | </pre> |
| + | |
| + | |
| | | |
| === Versions of the same product === | | === Versions of the same product === |
| * All versions of images selected have been processed | | * All versions of images selected have been processed |
− | * All images ever uploaded have been processed | + | * All images ever uploaded have been processed |
− | === Check complete products === | + | |
| + | |
| + | == How we can leverage the data == |
| + | |
| + | === Perform checks on products that are already complete === |
| * Check how similar ingredients lists are to what is detected | | * Check how similar ingredients lists are to what is detected |
| * Check if brands are suggested that are not here (eg Umbrella brands) | | * Check if brands are suggested that are not here (eg Umbrella brands) |
− | * Extract addresses | + | * Extract cooking instructions |
| + | * Extract storage information |
| + | * Extract non-essential marketing information (eg on wines) |
| + | * Extract postal addresses |
| * Extract URLs | | * Extract URLs |
| + | |
| | | |
| === Suggest values for incomplete products === | | === Suggest values for incomplete products === |
− | * Try to suggest possible categories based on extracted text
| |
− | * Try to suggest brands, both based on the Brand JSON field and on Raw text extraction
| |
| | | |
− | === Search engine (internal/external) considerations === | + | * Categories candidates, based on extracted text |
− | * Add raw output as hidden text to enable search engine to index incomplete products | + | |
| + | * Brands candidates based on: |
| + | ** the Brand JSON field |
| + | ** Raw text extraction |
| + | ** URL regex |
| + | *** Based on Wikidata urls |
| + | *** Based on urls extracted in other products, or set as Product urls |
| + | |
| + | * Packaging candidates |
| + | ** Taxonomy match |
| + | ** Existing values match |
| + | |
| + | * Location candidates |
| + | ** OpenStreetMap/Wikidata match |
| + | ** Taxonomy match |
| + | ** Existing values match |
| + | |
| + | * Packager codes candidates |
| + | ** EMB code |
| + | ** EU codes |
| + | |
| + | * Quantity candidates |
| + | ** Regex match |
| + | |
| + | * Labels, certifications, awards |
| + | ** Taxonomy match |
| + | ** Existing values match |
| + | |
| + | * Origin of ingredients |
| + | ** Taxonomy match + Dropdown |
| + | ** Existing values match + Dropdown |
| + | |
| + | * Manufacturing or processing places |
| + | ** OpenStreetMap/Wikidata match |
| + | ** Taxonomy match |
| + | ** Existing values match |
| + | |
| + | * Link to the product page on the official site of the producer |
| + | ** Regex match |
| + | ** Google Search with name+brand |
| + | |
| + | * Best before date |
| + | ** Regex match |
| + | |
| + | * City, state and country where purchased |
| + | ** OpenStreetMap/Wikidata match |
| + | ** Taxonomy match |
| + | ** Existing values match |
| + | |
| + | * Stores |
| + | ** Taxonomy match |
| + | ** Existing values match |
| + | |
| + | * Countries where sold |
| + | ** OpenStreetMap/Wikidata match |
| + | ** Taxonomy match |
| + | ** Existing values match |
| + | |
| + | * Ingredients list |
| + | ** Regex match |
| + | ** Taxonomy match |
| + | ** Existing values match |
| + | |
| + | * Traces |
| + | ** Regex match |
| + | ** Taxonomy match |
| + | ** Existing values match |
| + | |
| + | * Serving size |
| + | ** Regex match |
| + | |
| + | * Nutrients volume |
| + | ** Regex match |
| + | |
| + | * Ecological footprint |
| + | ** Regex match |
| + | |
| + | === Make uncomplete products searchable === |
| + | * Add raw JSON output as hidden by default text to enable search engines to index incomplete products (it could be collapsed and shown after the click of a button) |
| * Add hidden field to enable the internal search engine to find incomplete products | | * Add hidden field to enable the internal search engine to find incomplete products |
− | === New fields to add ? === | + | |
| + | === Expand Open Food Facts coverage by adding additional information in new fields === |
| + | * Description |
| + | ** some products are very talkative |
| + | ** it can be a first step before we add the other fields |
| * Cooking recommandation | | * Cooking recommandation |
| * Storage recommandation | | * Storage recommandation |
− | * Description (some products are very talkative)
| |
| * Postal address of the producer | | * Postal address of the producer |
| * Postal address of the distributor | | * Postal address of the distributor |
− | * Email Address | + | * Email Address of the producer |
− | * Phone number | + | * Email Address of the distributor |
− | * Website of the brand (vs product) | + | * Phone number of the producer |
| + | * Phone number of the distributor |
| + | * Website of the brand |
| * Ingredient percentage | | * Ingredient percentage |
| | | |