Changes

Jump to navigation Jump to search
2,024 bytes added ,  21:52, 14 February 2019
no edit summary
Line 49: Line 49:  
== Instructions ==
 
== Instructions ==
 
=== How to process images to generate JSONs ===
 
=== How to process images to generate JSONs ===
 +
 +
Needs to be executed in home/off/html/
 +
 
<pre>
 
<pre>
sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[0-9].jpg*"); do sudo python /home/xxxxxxxx.py $f;done'
+
sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[0-9]+.jpg*"); do sudo python /home/teolemon/ocr/cloudvision4.py $f;done'
sudo nohup sh -c 'for f in $(find . -name 'nutrition.*.full.jpg' -type f); do sudo python /home/xxxxxxxx.py $f;done'
+
sudo nohup sh -c 'for f in $(find . -name 'nutrition.*.full.jpg' -type f); do sudo python /home/teolemon/ocr/cloudvision4.py $f;done'
sudo nohup sh -c 'for f in $(find . -name 'front.*.full.jpg' -type f); do sudo python /home/xxxxxxxx.py $f;done'
+
sudo nohup sh -c 'for f in $(find . -name 'front.*.full.jpg' -type f); do sudo python /home/teolemon/ocr/cloudvision4.py $f;done'
sudo nohup sh -c 'for f in $(find . -name 'ingredients.*.full.jpg' -type f); do sudo python /home/xxxxxxxx.py $f;done'
+
sudo nohup sh -c 'for f in $(find . -name 'ingredients.*.full.jpg' -type f); do sudo python /home/teolemon/ocr/cloudvision4.py $f;done'
 +
</pre>
 +
 
 +
<pre>
 +
sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[1-1].jpg*"); do sudo python /home/teolemon/ocr/cloudvision4.py $f;done'
 +
sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[2-2].jpg*"); do sudo python /home/teolemon/ocr/cloudvision4.py $f;done'
 +
sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[3-3].jpg*"); do sudo python /home/teolemon/ocr/cloudvision4.py $f;done'
 +
sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[4-4].jpg*"); do sudo python /home/teolemon/ocr/cloudvision4.py $f;done'
 +
sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[5-5].jpg*"); do sudo python /home/teolemon/ocr/cloudvision4.py $f;done'
 +
sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[6-6].jpg*"); do sudo python /home/teolemon/ocr/cloudvision4.py $f;done'
 +
sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[7-7].jpg*"); do sudo python /home/teolemon/ocr/cloudvision4.py $f;done'
 
</pre>
 
</pre>
   Line 60: Line 73:  
46715+128832+59961+180420 = 415928 (edited)
 
46715+128832+59961+180420 = 415928 (edited)
 
<pre>
 
<pre>
teolemon@ks3095298:/home/off/html/images/products$ ls -lR /home/off/html/images/products/ | find . -name 'nutrition.*.full.jpg' -type f | wc -l
+
ls -lR /srv/off/html/images/products/ | find . -name 'nutrition.*.full.jpg' -type f | wc -l
 
46715
 
46715
teolemon@ks3095298:/home/off/html/images/products$ ls -lR /home/off/html/images/products/ | find . -name 'front.*.full.jpg' -type f | wc -l
+
ls -lR /srv/off/html/images/products/ | find . -name 'front.*.full.jpg' -type f | wc -l
 
128382
 
128382
teolemon@ks3095298:/home/off/html/images/products$ ls -lR /home/off/html/images/products/ | find . -name 'ingredients.*.full.jpg' -type f | wc -l
+
ls -lR /srv/off/html/images/products/ | find . -name 'ingredients.*.full.jpg' -type f | wc -l
 
59961
 
59961
teolemon@ks3095298:/home/off/html/images/products$ ls -lR /home/off/html/images/products/ | find . -type f -regex ".​​*/[0-9].jpg*​​" | wc -l
+
ls -lR /srv/off/html/images/products/ | find . -type f -regex ".​​*/[0-9].jpg*​​" | wc -l
 
180420
 
180420
 
</pre>
 
</pre>
 +
 +
<pre>
 +
Nov 1 - 13:22 - Paris Time
 +
ls -lR /srv/off/html/images/products | find . -type f -regex ".*/[0-9].jpg*" | wc -l
 +
287119
 +
</pre>
 +
 
==== Number of JSONs====
 
==== Number of JSONs====
 
<pre>
 
<pre>
ls -lR /home/off/html/images/products/ | find . -type f -regex ".​​*/[0-9].json*​​" | wc -l
+
ls -lR /srv/off/html/images/products/ | find . -type f -regex ".​​*/[0-9].json*​​" | wc -l
 
415308
 
415308
 +
</pre>
 +
 +
 +
<pre>
 +
Nov 1 2016 - 13:30
 +
teolemon@ns3362784:/srv/off/html/images/products$ ls -lR /srv/off/html/images/products | find . -type f -regex ".*/[0-9].json*" | wc -l
 +
253322
 
</pre>
 
</pre>
   Line 92: Line 119:  
</pre>
 
</pre>
    +
==== Search for a string in the generated JSONs ====
 +
grep --include=*.json -rnwl '/srv/off/html/images/products/' -e "halloween"
    
== Status ==
 
== Status ==
Line 97: Line 126:  
* remove json with error messages
 
* remove json with error messages
   −
== Open Food Facts ==
+
=== Open Food Facts ===
 
* raw images
 
* raw images
 
** existing
 
** existing
Line 110: Line 139:  
** existing
 
** existing
 
** processed
 
** processed
== Open Beauty Facts ==
+
 
 +
=== Open Beauty Facts ===
 
* raw images
 
* raw images
 
** existing
 
** existing
Line 128: Line 158:  
=== Perform checks on products that are already complete ===
 
=== Perform checks on products that are already complete ===
 
* Check how similar ingredients lists are to what is detected
 
* Check how similar ingredients lists are to what is detected
 +
** Calculate percentage of ingredients similarity
 +
*** List of parsed in ingredients list
 +
*** Ingredients found in the JSON (from the list of ingredients)
 +
 +
* Check for ingredients not found in the current ingredients list (if not empty)
 +
** Needs ingredients taxonomisation to avoid false positives
 +
 
* Check if brands are suggested that are not here (eg Umbrella brands)
 
* Check if brands are suggested that are not here (eg Umbrella brands)
* Extract cooking instructions
+
** Taxonomize brands
* Extract storage information
+
** Propose candidates
* Extract non-essential marketing information (eg on wines)
+
 
* Extract postal addresses
+
* Check nutrition values on new images
* Extract URLs
+
** Marcel Salathé seems to be working on a machine learning solution to extract nutrition values from raw images
    +
* Look for new languages
 +
** Generate a list of languages in each image/product
 +
*** Use language detection libs to detect all languages
 +
*** Use language value in JSON
 +
** Check if the product has images and values for all of them
 +
*** values that change are (front/ingredients/nutrition images) &  (front/ingredients text)
 +
** Make a page with a list of products with new languages to add
    
=== Suggest values for incomplete products ===
 
=== Suggest values for incomplete products ===
Line 222: Line 266:  
=== Expand Open Food Facts coverage by adding additional information in new fields ===
 
=== Expand Open Food Facts coverage by adding additional information in new fields ===
 
* Description  
 
* Description  
** some products are very talkative
+
** some products are very talkative, non-essential marketing information (eg on wines)
 
** it can be a first step before we add the other fields
 
** it can be a first step before we add the other fields
 
* Cooking recommandation
 
* Cooking recommandation

Navigation menu