Changes

Jump to navigation Jump to search
no edit summary
Line 19: Line 19:  
== Data sets ==
 
== Data sets ==
   −
* https://github.com/openfoodfacts/openfoodfacts-ai/blob/master/data-sets.md
+
* [https://github.com/openfoodfacts/openfoodfacts-ai/blob/master/data-sets.md#nutrition-tables-cropping-and-nutrition-facts-extraction https://github.com/openfoodfacts/openfoodfacts-ai/blob/master/data-sets.md]
   −
= How the model fits in the OFF infrastructure =
+
= How the model fits in the Open Food Facts infrastructure =
    
[[File:Nutrition Facts extraction flow.svg]]
 
[[File:Nutrition Facts extraction flow.svg]]
Line 41: Line 41:     
* Scoring des méthodes de détection de tableaux nutritionnels [https://drive.google.com/open?id=1-bo0v7eoRMtemZjcmqVzW0-0YMZaSOkB PDF] [https://drive.google.com/open?id=1KJ6H1HqHSHVBk3mWjEZZLLpcCt5whe6N docx]
 
* Scoring des méthodes de détection de tableaux nutritionnels [https://drive.google.com/open?id=1-bo0v7eoRMtemZjcmqVzW0-0YMZaSOkB PDF] [https://drive.google.com/open?id=1KJ6H1HqHSHVBk3mWjEZZLLpcCt5whe6N docx]
* Analyse approfondie sur la détection
+
* Analyse approfondie sur la détection de tableaux nutritionnels [https://drive.google.com/open?id=1nPxOGFgjKaKuUcnVC0nbNAijt1al9rvo pdf]
de tableaux nutritionnels [https://drive.google.com/open?id=1nPxOGFgjKaKuUcnVC0nbNAijt1al9rvo pdf]
      
* Table extraction (V1): https://github.com/openfoodfacts/off-nutrition-table-extractor. Mainly focus on the nutritional table detection part, the extraction process remains quite simple.
 
* Table extraction (V1): https://github.com/openfoodfacts/off-nutrition-table-extractor. Mainly focus on the nutritional table detection part, the extraction process remains quite simple.
 
* Table extraction (v2): https://github.com/cgandon/openfoodfacts-nutriments
 
* Table extraction (v2): https://github.com/cgandon/openfoodfacts-nutriments
 +
* Work by Sadok and Yichen as part of the Microsoft ShareAI program
 +
** TableNet
 +
** GraphNet
    
= Other approaches =
 
= Other approaches =
Line 63: Line 65:     
* Test of an end-to-end model, through an API as a starter: Form Recognizer or Google Document AI API. If results are promising, we develop in-house end-to-end model, otherwise a layout model.
 
* Test of an end-to-end model, through an API as a starter: Form Recognizer or Google Document AI API. If results are promising, we develop in-house end-to-end model, otherwise a layout model.
* Thourough litterature review
+
* Thorough literature review
 
* Architecture choice
 
* Architecture choice
 
* Annotation campaign
 
* Annotation campaign
Line 72: Line 74:     
Yichen: test of image preprocessing before Form Recognizer (layout model), test "manual" approaches.
 
Yichen: test of image preprocessing before Form Recognizer (layout model), test "manual" approaches.
 +
* tested a pre-processing method (ported from Mathematica to python)
 +
* OCR results worsen after binarization
 +
* testing method to morph deformed images (by detecting horizontal and vertical lines)
 +
* "manual" approaches: compare positions of bounding boxes of OCR, use borders if we have some
 +
** manual horizontal or vertical scans of bounding boxes, using the angles of bounding boxes
 +
** robotoff has python classes for importing/analyzing Google Cloud Vision
   −
Raphaël: Litterature review (to be continued)
+
Raphaël: Literature review (to be continued)
 +
 
 +
= Notes 24/07 =
 +
 
 +
* Ramzi: tested end-to-end model with learning
 +
** missing one step, results expected next week
 +
* Yasmine + Yichen: working on pre-processing
 +
** Testing cylinders distortions -> flat rectangle
 +
*** open cv
 +
*** Tesseract performance very poor
 +
 
 +
Next week:
 +
* Ramzi
 +
** finish test end-to-end model
 +
** increase size of data set to get significant results
 +
* Yasmine + Yichen
 +
** Cylinder unwrapping (classic + deep learning models)
 +
*** review litterature
 +
*** create data set for Ramzi using paid API "perfect label"
 +
** table structure detection : associate bounding boxes to key value pairs
 +
* All
 +
** Create test set with cylindric photos + unwrapped photos
    
= Resources =
 
= Resources =
Line 82: Line 111:  
** Step 2: Download Lobe - For Mac: https://aka.ms/DownloadLobeMac - For PC: https://aka.ms/DownloadLobeWindows
 
** Step 2: Download Lobe - For Mac: https://aka.ms/DownloadLobeMac - For PC: https://aka.ms/DownloadLobeWindows
 
* [https://flow.microsoft.com/en-us/ai-builder/ AI Builder] - import your photos right away into a Power App (or a Power Automate flow) so you can use the output in any real world application.
 
* [https://flow.microsoft.com/en-us/ai-builder/ AI Builder] - import your photos right away into a Power App (or a Power Automate flow) so you can use the output in any real world application.
 +
 +
= Quality metrics =
 +
* [https://docs.google.com/document/d/1sJNZOq2Gs6gnNSCV2hPlnl4-3Z_SdQFQuorqoRw1cjA/edit Quality Metrics & Golden sets]

Navigation menu