Difference between revisions of "OCR/Results"

From Open Food Facts wiki
Jump to navigation Jump to search
Line 1: Line 1:
How to process the 1.4 million JSON files we've generated
+
How to process the 400k JSON files we've generated
  
 
== Example ==
 
== Example ==
Line 48: Line 48:
  
 
== What to do ==
 
== What to do ==
 +
=== TODO ===
 +
* remove json with error messages
 +
* move all the JSONs to production
 +
=== Results ===
 +
==== Number of images ====
 +
46715+128832+59961+180420 = 415928(edited)
 +
<pre>
 +
teolemon@ks3095298:/home/off/html/images/products$ ls -lR /home/off/html/images/products/ | find . -name 'nutrition.*.full.jpg' -type f | wc -l
 +
46715
 +
teolemon@ks3095298:/home/off/html/images/products$ ls -lR /home/off/html/images/products/ | find . -name 'front.*.full.jpg' -type f | wc -l
 +
128382
 +
teolemon@ks3095298:/home/off/html/images/products$ ls -lR /home/off/html/images/products/ | find . -name 'ingredients.*.full.jpg' -type f | wc -l
 +
59961
 +
teolemon@ks3095298:/home/off/html/images/products$ ls -lR /home/off/html/images/products/ | find . -type f -regex ".​​*/[0-9].jpg*​​" | wc -l
 +
180420
 +
</pre>
 +
==== Number of JSONs====
 +
<pre>
 +
415308
 +
</pre>
 
=== Versions of the same product ===
 
=== Versions of the same product ===
 
* All versions of images selected have been processed
 
* All versions of images selected have been processed

Revision as of 16:49, 30 January 2016

How to process the 400k JSON files we've generated

Example

Image

ingredients.12.400.jpg

Output

http://static.openfoodfacts.ovh/images/products/40193045/ingredients.12.full.json

{  
   u'responses':[  
      {  
         u'textAnnotations':[  
            {  
               u'locale':u'de',
               u'description':               u'Doppelrahmstufe\nAT Kasezubereitung 60% F.LT\nzutaten:Frischkase,
               2               gefriergetrockne\nter Gorgonzola,
               Salz,
               Verdickungsmittet\nJohannisbrotkernmehl und Guarkemmehlt\nAroma,
               Stickstoff zum Aufschlagen\nNach dem Offnen alsbald verzehren\nUngeoffnet gekuhlt mindestens\nhaltbar bis:siehe Boden\nBY 706\nIn Deutschland hergestellt\n',
               u'boundingPoly':{  
                  u'vertices':[  
                     {  
                        u'y':232,
                        u'x':190
                     },
                     {  
                        u'y':232,
                        u'x':1500
                     },
                     {  
                        u'y':1410,
                        u'x':1500
                     },
                     {  
                        u'y':1410,
                        u'x':190
                     }
                  ]
               }
            }
         ]
      }
   ]
}

Comments

What to do

TODO

  • remove json with error messages
  • move all the JSONs to production

Results

Number of images

46715+128832+59961+180420 = 415928(edited)

teolemon@ks3095298:/home/off/html/images/products$ ls -lR /home/off/html/images/products/ | find . -name 'nutrition.*.full.jpg' -type f | wc -l
46715
teolemon@ks3095298:/home/off/html/images/products$ ls -lR /home/off/html/images/products/ | find . -name 'front.*.full.jpg' -type f | wc -l
128382
teolemon@ks3095298:/home/off/html/images/products$ ls -lR /home/off/html/images/products/ | find . -name 'ingredients.*.full.jpg' -type f | wc -l
59961
teolemon@ks3095298:/home/off/html/images/products$ ls -lR /home/off/html/images/products/ | find . -type f -regex ".​​*/[0-9].jpg*​​" | wc -l
180420

Number of JSONs

415308

Versions of the same product

  • All versions of images selected have been processed
  • All images ever uploaded have been processed

Check complete products

  • Check how similar ingredients lists are to what is detected
  • Check if brands are suggested that are not here (eg Umbrella brands)
  • Extract addresses
  • Extract URLs

Suggest values for incomplete products

  • Try to suggest possible categories based on extracted text
  • Try to suggest brands, both based on the Brand JSON field and on Raw text extraction

Search engine (internal/external) considerations

  • Add raw output as hidden text to enable search engine to index incomplete products
  • Add hidden field to enable the internal search engine to find incomplete products

New fields to add ?

  • Cooking recommandation
  • Storage recommandation
  • Description (some products are very talkative)
  • Postal address of the producer
  • Postal address of the distributor
  • Email Address
  • Phone number
  • Website of the brand (vs product)
  • Ingredient percentage

Kind of images processed

Cropped front images

Cropped ingredient images

Cropped nutrition images

Raw images