Changes

Jump to navigation Jump to search
no edit summary
Line 19: Line 19:  
== Data sets ==
 
== Data sets ==
   −
* https://github.com/openfoodfacts/openfoodfacts-ai/blob/master/data-sets.md
+
* [https://github.com/openfoodfacts/openfoodfacts-ai/blob/master/data-sets.md#nutrition-tables-cropping-and-nutrition-facts-extraction https://github.com/openfoodfacts/openfoodfacts-ai/blob/master/data-sets.md]
   −
= How the model fits in the OFF infrastructure =
+
= How the model fits in the Open Food Facts infrastructure =
    
[[File:Nutrition Facts extraction flow.svg]]
 
[[File:Nutrition Facts extraction flow.svg]]
Line 41: Line 41:     
* Scoring des méthodes de détection de tableaux nutritionnels [https://drive.google.com/open?id=1-bo0v7eoRMtemZjcmqVzW0-0YMZaSOkB PDF] [https://drive.google.com/open?id=1KJ6H1HqHSHVBk3mWjEZZLLpcCt5whe6N docx]
 
* Scoring des méthodes de détection de tableaux nutritionnels [https://drive.google.com/open?id=1-bo0v7eoRMtemZjcmqVzW0-0YMZaSOkB PDF] [https://drive.google.com/open?id=1KJ6H1HqHSHVBk3mWjEZZLLpcCt5whe6N docx]
* Analyse approfondie sur la détection
+
* Analyse approfondie sur la détection de tableaux nutritionnels [https://drive.google.com/open?id=1nPxOGFgjKaKuUcnVC0nbNAijt1al9rvo pdf]
de tableaux nutritionnels [https://drive.google.com/open?id=1nPxOGFgjKaKuUcnVC0nbNAijt1al9rvo pdf]
      
* Table extraction (V1): https://github.com/openfoodfacts/off-nutrition-table-extractor. Mainly focus on the nutritional table detection part, the extraction process remains quite simple.
 
* Table extraction (V1): https://github.com/openfoodfacts/off-nutrition-table-extractor. Mainly focus on the nutritional table detection part, the extraction process remains quite simple.
 
* Table extraction (v2): https://github.com/cgandon/openfoodfacts-nutriments
 
* Table extraction (v2): https://github.com/cgandon/openfoodfacts-nutriments
 +
* Work by Sadok and Yichen as part of the Microsoft ShareAI program
 +
** TableNet
 +
** GraphNet
    
= Other approaches =
 
= Other approaches =
Line 55: Line 57:     
Each approach requires different kinds of annotation, so chosing the most effective approach at the beginning of the project is crucial.
 
Each approach requires different kinds of annotation, so chosing the most effective approach at the beginning of the project is crucial.
 +
 +
[[File:Nutrition Facts extraction approaches (1).png]]
 +
 +
Source: https://docs.google.com/drawings/d/1YKgnTEX1RBgsMQpO4JIt94sMwf1IC_fPWIvPnUjswTg/edit?usp=sharing
    
= Planning =
 
= Planning =
    
* Test of an end-to-end model, through an API as a starter: Form Recognizer or Google Document AI API. If results are promising, we develop in-house end-to-end model, otherwise a layout model.
 
* Test of an end-to-end model, through an API as a starter: Form Recognizer or Google Document AI API. If results are promising, we develop in-house end-to-end model, otherwise a layout model.
* Thourough litterature review
+
* Thorough literature review
 
* Architecture choice
 
* Architecture choice
 
* Annotation campaign
 
* Annotation campaign
 
* Experimentation: model training and scoring
 
* Experimentation: model training and scoring
 
* Integration to Robotoff/Hunger Games
 
* Integration to Robotoff/Hunger Games
 +
 +
Ramzi: test of end-to-end model with Form Recognizer API.
 +
 +
Yichen: test of image preprocessing before Form Recognizer (layout model), test "manual" approaches.
 +
* tested a pre-processing method (ported from Mathematica to python)
 +
* OCR results worsen after binarization
 +
* testing method to morph deformed images (by detecting horizontal and vertical lines)
 +
* "manual" approaches: compare positions of bounding boxes of OCR, use borders if we have some
 +
** manual horizontal or vertical scans of bounding boxes, using the angles of bounding boxes
 +
** robotoff has python classes for importing/analyzing Google Cloud Vision
 +
 +
Raphaël: Literature review (to be continued)
 +
 +
= Notes 24/07 =
 +
 +
* Ramzi: tested end-to-end model with learning
 +
** missing one step, results expected next week
 +
* Yasmine + Yichen: working on pre-processing
 +
** Testing cylinders distortions -> flat rectangle
 +
*** open cv
 +
*** Tesseract performance very poor
 +
 +
Next week:
 +
* Ramzi
 +
** finish test end-to-end model
 +
** increase size of data set to get significant results
 +
* Yasmine + Yichen
 +
** Cylinder unwrapping (classic + deep learning models)
 +
*** review litterature
 +
*** create data set for Ramzi using paid API "perfect label"
 +
** table structure detection : associate bounding boxes to key value pairs
 +
* All
 +
** Create test set with cylindric photos + unwrapped photos
    
= Resources =
 
= Resources =
Line 72: Line 111:  
** Step 2: Download Lobe - For Mac: https://aka.ms/DownloadLobeMac - For PC: https://aka.ms/DownloadLobeWindows
 
** Step 2: Download Lobe - For Mac: https://aka.ms/DownloadLobeMac - For PC: https://aka.ms/DownloadLobeWindows
 
* [https://flow.microsoft.com/en-us/ai-builder/ AI Builder] - import your photos right away into a Power App (or a Power Automate flow) so you can use the output in any real world application.
 
* [https://flow.microsoft.com/en-us/ai-builder/ AI Builder] - import your photos right away into a Power App (or a Power Automate flow) so you can use the output in any real world application.
 +
 +
= Quality metrics =
 +
* [https://docs.google.com/document/d/1sJNZOq2Gs6gnNSCV2hPlnl4-3Z_SdQFQuorqoRw1cjA/edit Quality Metrics & Golden sets]

Navigation menu