OCR/Results

How to process the 400k JSON files we've generated

Example

Image

Output

http://static.openfoodfacts.ovh/images/products/40193045/ingredients.12.full.json

{  
   u'responses':[  
      {  
         u'textAnnotations':[  
            {  
               u'locale':u'de',
               u'description':               u'Doppelrahmstufe\nAT Kasezubereitung 60% F.LT\nzutaten:Frischkase,
               2               gefriergetrockne\nter Gorgonzola,
               Salz,
               Verdickungsmittet\nJohannisbrotkernmehl und Guarkemmehlt\nAroma,
               Stickstoff zum Aufschlagen\nNach dem Offnen alsbald verzehren\nUngeoffnet gekuhlt mindestens\nhaltbar bis:siehe Boden\nBY 706\nIn Deutschland hergestellt\n',
               u'boundingPoly':{  
                  u'vertices':[  
                     {  
                        u'y':232,
                        u'x':190
                     },
                     {  
                        u'y':232,
                        u'x':1500
                     },
                     {  
                        u'y':1410,
                        u'x':1500
                     },
                     {  
                        u'y':1410,
                        u'x':190
                     }
                  ]
               }
            }
         ]
      }
   ]
}

Instructions

How to process images to generate JSONs

Needs to be executed in home/off/html/

sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[0-9].jpg*"); do sudo python /home/teolemon/ocr/cloudvision4.py $f;done'
sudo nohup sh -c 'for f in $(find . -name 'nutrition.*.full.jpg' -type f); do sudo python /home/teolemon/ocr/cloudvision4.py $f;done'
sudo nohup sh -c 'for f in $(find . -name 'front.*.full.jpg' -type f); do sudo python /home/teolemon/ocr/cloudvision4.py $f;done'
sudo nohup sh -c 'for f in $(find . -name 'ingredients.*.full.jpg' -type f); do sudo python /home/teolemon/ocr/cloudvision4.py $f;done'

sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[1-1].jpg*"); do sudo python /home/teolemon/ocr/cloudvision4.py $f;done'
sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[2-2].jpg*"); do sudo python /home/teolemon/ocr/cloudvision4.py $f;done'
sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[3-3].jpg*"); do sudo python /home/teolemon/ocr/cloudvision4.py $f;done'
sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[4-4].jpg*"); do sudo python /home/teolemon/ocr/cloudvision4.py $f;done'
sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[5-5].jpg*"); do sudo python /home/teolemon/ocr/cloudvision4.py $f;done'
sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[6-6].jpg*"); do sudo python /home/teolemon/ocr/cloudvision4.py $f;done'
sudo nohup sh -c 'for f in $(find . -type f -regex ".*/[7-7].jpg*"); do sudo python /home/teolemon/ocr/cloudvision4.py $f;done'

How to check the results

Number of images

46715+128832+59961+180420 = 415928 (edited)

ls -lR /srv/off/html/images/products/ | find . -name 'nutrition.*.full.jpg' -type f | wc -l
46715
ls -lR /srv/off/html/images/products/ | find . -name 'front.*.full.jpg' -type f | wc -l
128382
ls -lR /srv/off/html/images/products/ | find . -name 'ingredients.*.full.jpg' -type f | wc -l
59961
ls -lR /srv/off/html/images/products/ | find . -type f -regex ".​​*/[0-9].jpg*​​" | wc -l
180420

Nov 1 - 13:22 - Paris Time
ls -lR /srv/off/html/images/products | find . -type f -regex ".*/[0-9].jpg*" | wc -l
287119

Number of JSONs

ls -lR /srv/off/html/images/products/ | find . -type f -regex ".​​*/[0-9].json*​​" | wc -l
415308

Nov 1 2016 - 13:30
teolemon@ns3362784:/srv/off/html/images/products$ ls -lR /srv/off/html/images/products | find . -type f -regex ".*/[0-9].json*" | wc -l
253322

Number of files in the directory

teolemon@ns3362784:/home/off/html/images/products$ find . -type f | sed 's/.*\.//' | sort | uniq -c
   2514 front
     35 gif
     27 ingredients
2890282 jpg
 478982 json
      1 kxs2sl
      1 out
   3311 png
    210 tif
    153 txt

Search for a string in the generated JSONs

grep --include=*.json -rnwl '/srv/off/html/images/products/' -e "halloween"

Status

TODO

• remove json with error messages

Open Food Facts

• raw images
  • existing
  • processed
• front
  • existing
  • processed
• ingredients
  • existing
  • processed
• nutrition
  • existing
  • processed

Open Beauty Facts

• raw images
  • existing
  • processed
• front
  • existing
  • processed
• ingredients
  • existing
  • processed
• nutrition
  • existing
  • processed

How we can leverage the data

Perform checks on products that are already complete

• Check how similar ingredients lists are to what is detected
  • Calculate percentage of ingredients similarity
    • List of parsed in ingredients list
    • Ingredients found in the JSON (from the list of ingredients)

• Check for ingredients not found in the current ingredients list (if not empty)
  • Needs ingredients taxonomisation to avoid false positives

• Check if brands are suggested that are not here (eg Umbrella brands)
  • Taxonomize brands
  • Propose candidates

• Check nutrition values on new images
  • Marcel Salathé seems to be working on a machine learning solution to extract nutrition values from raw images

• Look for new languages
  • Generate a list of languages in each image/product
    • Use language detection libs to detect all languages
    • Use language value in JSON
  • Check if the product has images and values for all of them
    • values that change are (front/ingredients/nutrition images) & (front/ingredients text)
  • Make a page with a list of products with new languages to add

Suggest values for incomplete products

• Categories candidates, based on extracted text

• Brands candidates based on:
  • the Brand JSON field
  • Raw text extraction
  • URL regex
    • Based on Wikidata urls
    • Based on urls extracted in other products, or set as Product urls

• Packaging candidates
  • Taxonomy match
  • Existing values match

• Location candidates
  • OpenStreetMap/Wikidata match
  • Taxonomy match
  • Existing values match

• Packager codes candidates
  • EMB code
  • EU codes

• Quantity candidates
  • Regex match

• Labels, certifications, awards
  • Taxonomy match
  • Existing values match

• Origin of ingredients
  • Taxonomy match + Dropdown
  • Existing values match + Dropdown

• Manufacturing or processing places
  • OpenStreetMap/Wikidata match
  • Taxonomy match
  • Existing values match

• Link to the product page on the official site of the producer
  • Regex match
  • Google Search with name+brand

• Best before date
  • Regex match

• City, state and country where purchased
  • OpenStreetMap/Wikidata match
  • Taxonomy match
  • Existing values match

• Stores
  • Taxonomy match
  • Existing values match

• Countries where sold
  • OpenStreetMap/Wikidata match
  • Taxonomy match
  • Existing values match

• Ingredients list
  • Regex match
  • Taxonomy match
  • Existing values match

• Traces
  • Regex match
  • Taxonomy match
  • Existing values match

• Serving size
  • Regex match

• Nutrients volume
  • Regex match

• Ecological footprint
  • Regex match

Make uncomplete products searchable

• Add raw JSON output as hidden by default text to enable search engines to index incomplete products (it could be collapsed and shown after the click of a button)
• Add hidden field to enable the internal search engine to find incomplete products

Expand Open Food Facts coverage by adding additional information in new fields

• Description
  • some products are very talkative, non-essential marketing information (eg on wines)
  • it can be a first step before we add the other fields
• Cooking recommandation
• Storage recommandation
• Postal address of the producer
• Postal address of the distributor
• Email Address of the producer
• Email Address of the distributor
• Phone number of the producer
• Phone number of the distributor
• Website of the brand
• Ingredient percentage