Line 49: |
Line 49: |
| == Instructions == | | == Instructions == |
| === How to process images to generate JSONs === | | === How to process images to generate JSONs === |
| + | |
| + | Needs to be executed in home/off/html/ |
| + | |
| + | <pre> |
| + | sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[0-9]+.jpg*"); do sudo python /home/teolemon/ocr/cloudvision4.py $f;done' |
| + | sudo nohup sh -c 'for f in $(find . -name 'nutrition.*.full.jpg' -type f); do sudo python /home/teolemon/ocr/cloudvision4.py $f;done' |
| + | sudo nohup sh -c 'for f in $(find . -name 'front.*.full.jpg' -type f); do sudo python /home/teolemon/ocr/cloudvision4.py $f;done' |
| + | sudo nohup sh -c 'for f in $(find . -name 'ingredients.*.full.jpg' -type f); do sudo python /home/teolemon/ocr/cloudvision4.py $f;done' |
| + | </pre> |
| + | |
| <pre> | | <pre> |
− | sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[0-9].jpg*"); do sudo python /home/xxxxxxxx.py $f;done' | + | sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[1-1].jpg*"); do sudo python /home/teolemon/ocr/cloudvision4.py $f;done' |
− | sudo nohup sh -c 'for f in $(find . -name 'nutrition.*.full.jpg' -type f); do sudo python /home/xxxxxxxx.py $f;done' | + | sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[2-2].jpg*"); do sudo python /home/teolemon/ocr/cloudvision4.py $f;done' |
− | sudo nohup sh -c 'for f in $(find . -name 'front.*.full.jpg' -type f); do sudo python /home/xxxxxxxx.py $f;done' | + | sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[3-3].jpg*"); do sudo python /home/teolemon/ocr/cloudvision4.py $f;done' |
− | sudo nohup sh -c 'for f in $(find . -name 'ingredients.*.full.jpg' -type f); do sudo python /home/xxxxxxxx.py $f;done' | + | sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[4-4].jpg*"); do sudo python /home/teolemon/ocr/cloudvision4.py $f;done' |
| + | sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[5-5].jpg*"); do sudo python /home/teolemon/ocr/cloudvision4.py $f;done' |
| + | sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[6-6].jpg*"); do sudo python /home/teolemon/ocr/cloudvision4.py $f;done' |
| + | sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[7-7].jpg*"); do sudo python /home/teolemon/ocr/cloudvision4.py $f;done' |
| </pre> | | </pre> |
| | | |
Line 60: |
Line 73: |
| 46715+128832+59961+180420 = 415928 (edited) | | 46715+128832+59961+180420 = 415928 (edited) |
| <pre> | | <pre> |
− | teolemon@ks3095298:/home/off/html/images/products$ ls -lR /home/off/html/images/products/ | find . -name 'nutrition.*.full.jpg' -type f | wc -l
| + | ls -lR /srv/off/html/images/products/ | find . -name 'nutrition.*.full.jpg' -type f | wc -l |
| 46715 | | 46715 |
− | teolemon@ks3095298:/home/off/html/images/products$ ls -lR /home/off/html/images/products/ | find . -name 'front.*.full.jpg' -type f | wc -l
| + | ls -lR /srv/off/html/images/products/ | find . -name 'front.*.full.jpg' -type f | wc -l |
| 128382 | | 128382 |
− | teolemon@ks3095298:/home/off/html/images/products$ ls -lR /home/off/html/images/products/ | find . -name 'ingredients.*.full.jpg' -type f | wc -l
| + | ls -lR /srv/off/html/images/products/ | find . -name 'ingredients.*.full.jpg' -type f | wc -l |
| 59961 | | 59961 |
− | teolemon@ks3095298:/home/off/html/images/products$ ls -lR /home/off/html/images/products/ | find . -type f -regex ".*/[0-9].jpg*" | wc -l
| + | ls -lR /srv/off/html/images/products/ | find . -type f -regex ".*/[0-9].jpg*" | wc -l |
| 180420 | | 180420 |
| </pre> | | </pre> |
| + | |
| + | <pre> |
| + | Nov 1 - 13:22 - Paris Time |
| + | ls -lR /srv/off/html/images/products | find . -type f -regex ".*/[0-9].jpg*" | wc -l |
| + | 287119 |
| + | </pre> |
| + | |
| ==== Number of JSONs==== | | ==== Number of JSONs==== |
| <pre> | | <pre> |
− | ls -lR /home/off/html/images/products/ | find . -type f -regex ".*/[0-9].json*" | wc -l | + | ls -lR /srv/off/html/images/products/ | find . -type f -regex ".*/[0-9].json*" | wc -l |
| 415308 | | 415308 |
| + | </pre> |
| + | |
| + | |
| + | <pre> |
| + | Nov 1 2016 - 13:30 |
| + | teolemon@ns3362784:/srv/off/html/images/products$ ls -lR /srv/off/html/images/products | find . -type f -regex ".*/[0-9].json*" | wc -l |
| + | 253322 |
| </pre> | | </pre> |
| | | |
Line 92: |
Line 119: |
| </pre> | | </pre> |
| | | |
| + | ==== Search for a string in the generated JSONs ==== |
| + | grep --include=*.json -rnwl '/srv/off/html/images/products/' -e "halloween" |
| | | |
| == Status == | | == Status == |
Line 129: |
Line 158: |
| === Perform checks on products that are already complete === | | === Perform checks on products that are already complete === |
| * Check how similar ingredients lists are to what is detected | | * Check how similar ingredients lists are to what is detected |
| + | ** Calculate percentage of ingredients similarity |
| + | *** List of parsed in ingredients list |
| + | *** Ingredients found in the JSON (from the list of ingredients) |
| + | |
| + | * Check for ingredients not found in the current ingredients list (if not empty) |
| + | ** Needs ingredients taxonomisation to avoid false positives |
| + | |
| * Check if brands are suggested that are not here (eg Umbrella brands) | | * Check if brands are suggested that are not here (eg Umbrella brands) |
− | * Extract cooking instructions | + | ** Taxonomize brands |
− | * Extract storage information | + | ** Propose candidates |
− | * Extract non-essential marketing information (eg on wines) | + | |
− | * Extract postal addresses | + | * Check nutrition values on new images |
− | * Extract URLs | + | ** Marcel Salathé seems to be working on a machine learning solution to extract nutrition values from raw images |
| | | |
| + | * Look for new languages |
| + | ** Generate a list of languages in each image/product |
| + | *** Use language detection libs to detect all languages |
| + | *** Use language value in JSON |
| + | ** Check if the product has images and values for all of them |
| + | *** values that change are (front/ingredients/nutrition images) & (front/ingredients text) |
| + | ** Make a page with a list of products with new languages to add |
| | | |
| === Suggest values for incomplete products === | | === Suggest values for incomplete products === |
Line 223: |
Line 266: |
| === Expand Open Food Facts coverage by adding additional information in new fields === | | === Expand Open Food Facts coverage by adding additional information in new fields === |
| * Description | | * Description |
− | ** some products are very talkative | + | ** some products are very talkative, non-essential marketing information (eg on wines) |
| ** it can be a first step before we add the other fields | | ** it can be a first step before we add the other fields |
| * Cooking recommandation | | * Cooking recommandation |