Difference between revisions of "OCR/Results"

Revision as of 11:08, 1 November 2016

How to process the 400k JSON files we've generated

Example

Image

ingredients.12.400.jpg

Output

http://static.openfoodfacts.ovh/images/products/40193045/ingredients.12.full.json

{  
   u'responses':[  
      {  
         u'textAnnotations':[  
            {  
               u'locale':u'de',
               u'description':               u'Doppelrahmstufe\nAT Kasezubereitung 60% F.LT\nzutaten:Frischkase,
               2               gefriergetrockne\nter Gorgonzola,
               Salz,
               Verdickungsmittet\nJohannisbrotkernmehl und Guarkemmehlt\nAroma,
               Stickstoff zum Aufschlagen\nNach dem Offnen alsbald verzehren\nUngeoffnet gekuhlt mindestens\nhaltbar bis:siehe Boden\nBY 706\nIn Deutschland hergestellt\n',
               u'boundingPoly':{  
                  u'vertices':[  
                     {  
                        u'y':232,
                        u'x':190
                     },
                     {  
                        u'y':232,
                        u'x':1500
                     },
                     {  
                        u'y':1410,
                        u'x':1500
                     },
                     {  
                        u'y':1410,
                        u'x':190
                     }
                  ]
               }
            }
         ]
      }
   ]
}

Comments

Instructions

How to process images to generate JSONs

Needs to be executed in home/off/html/

sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[0-9].jpg*"); do sudo python /home/xxxxxxxx.py $f;done'
sudo nohup sh -c 'for f in $(find . -name 'nutrition.*.full.jpg' -type f); do sudo python /home/xxxxxxxx.py $f;done'
sudo nohup sh -c 'for f in $(find . -name 'front.*.full.jpg' -type f); do sudo python /home/xxxxxxxx.py $f;done'
sudo nohup sh -c 'for f in $(find . -name 'ingredients.*.full.jpg' -type f); do sudo python /home/xxxxxxxx.py $f;done'

sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[1-1].jpg*"); do sudo python /home/teolemon/cloudvision4.py $f;done'
sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[2-2].jpg*"); do sudo python /home/teolemon/cloudvision4.py $f;done'
sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[3-3].jpg*"); do sudo python /home/teolemon/cloudvision4.py $f;done'
sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[4-4].jpg*"); do sudo python /home/teolemon/cloudvision4.py $f;done'
sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[5-5].jpg*"); do sudo python /home/teolemon/cloudvision4.py $f;done'
sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[6-6].jpg*"); do sudo python /home/teolemon/cloudvision4.py $f;done'
sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[7-7].jpg*"); do sudo python /home/teolemon/cloudvision4.py $f;done'

How to check the results

Number of images

46715+128832+59961+180420 = 415928 (edited)

ls -lR /home/off/html/images/products/ | find . -name 'nutrition.*.full.jpg' -type f | wc -l
46715
ls -lR /home/off/html/images/products/ | find . -name 'front.*.full.jpg' -type f | wc -l
128382
ls -lR /home/off/html/images/products/ | find . -name 'ingredients.*.full.jpg' -type f | wc -l
59961
ls -lR /home/off/html/images/products/ | find . -type f -regex ".*/[0-9].jpg*" | wc -l
180420

Number of JSONs

ls -lR /home/off/html/images/products/ | find . -type f -regex ".*/[0-9].json*" | wc -l
415308

Number of files in the directory

teolemon@ns3362784:/home/off/html/images/products$ find . -type f | sed 's/.*\.//' | sort | uniq -c
   2514 front
     35 gif
     27 ingredients
2890282 jpg
 478982 json
      1 kxs2sl
      1 out
   3311 png
    210 tif
    153 txt

Search for a string in the generated JSONs

grep --include=*.json -rnwl '/home/off/html/images/products/' -e "halloween"

Status

TODO

remove json with error messages

Open Food Facts

raw images
- existing
- processed
front
- existing
- processed
ingredients
- existing
- processed
nutrition
- existing
- processed

Open Beauty Facts

raw images
- existing
- processed
front
- existing
- processed
ingredients
- existing
- processed
nutrition
- existing
- processed

How we can leverage the data

Perform checks on products that are already complete

Check how similar ingredients lists are to what is detected
- Calculate percentage of ingredients similarity
  - List of parsed in ingredients list
  - Ingredients found in the JSON (from the list of ingredients)

Check for ingredients not found in the current ingredients list (if not empty)
- Needs ingredients taxonomisation to avoid false positives

Check if brands are suggested that are not here (eg Umbrella brands)
- Taxonomize brands
- Propose candidates

Check nutrition values on new images
- Marcel Salathé seems to be working on a machine learning solution to extract nutrition values from raw images

Look for new languages
- Generate a list of languages in each image/product
  - Use language detection libs to detect all languages
  - Use language value in JSON
- Check if the product has images and values for all of them
  - values that change are (front/ingredients/nutrition images) & (front/ingredients text)
- Make a page with a list of products with new languages to add

Suggest values for incomplete products

Categories candidates, based on extracted text

Brands candidates based on:
- the Brand JSON field
- Raw text extraction
- URL regex
  - Based on Wikidata urls
  - Based on urls extracted in other products, or set as Product urls

Packaging candidates
- Taxonomy match
- Existing values match

Location candidates
- OpenStreetMap/Wikidata match
- Taxonomy match
- Existing values match

Packager codes candidates
- EMB code
- EU codes

Quantity candidates
- Regex match

Labels, certifications, awards
- Taxonomy match
- Existing values match

Origin of ingredients
- Taxonomy match + Dropdown
- Existing values match + Dropdown

Manufacturing or processing places
- OpenStreetMap/Wikidata match
- Taxonomy match
- Existing values match

Link to the product page on the official site of the producer
- Regex match
- Google Search with name+brand

Best before date
- Regex match

City, state and country where purchased
- OpenStreetMap/Wikidata match
- Taxonomy match
- Existing values match

Stores
- Taxonomy match
- Existing values match

Countries where sold
- OpenStreetMap/Wikidata match
- Taxonomy match
- Existing values match

Ingredients list
- Regex match
- Taxonomy match
- Existing values match

Traces
- Regex match
- Taxonomy match
- Existing values match

Serving size
- Regex match

Nutrients volume
- Regex match

Ecological footprint
- Regex match

Make uncomplete products searchable

Add raw JSON output as hidden by default text to enable search engines to index incomplete products (it could be collapsed and shown after the click of a button)
Add hidden field to enable the internal search engine to find incomplete products

Expand Open Food Facts coverage by adding additional information in new fields

Description
- some products are very talkative, non-essential marketing information (eg on wines)
- it can be a first step before we add the other fields
Cooking recommandation
Storage recommandation
Postal address of the producer
Postal address of the distributor
Email Address of the producer
Email Address of the distributor
Phone number of the producer
Phone number of the distributor
Website of the brand
Ingredient percentage

Kind of images processed

Cropped front images

Cropped ingredient images

Cropped nutrition images

Raw images