Line 19: |
Line 19: |
| == Data sets == | | == Data sets == |
| | | |
− | * https://github.com/openfoodfacts/openfoodfacts-ai/blob/master/data-sets.md | + | * [https://github.com/openfoodfacts/openfoodfacts-ai/blob/master/data-sets.md#nutrition-tables-cropping-and-nutrition-facts-extraction https://github.com/openfoodfacts/openfoodfacts-ai/blob/master/data-sets.md] |
| | | |
− | = How the model fits in the OFF infrastructure = | + | = How the model fits in the Open Food Facts infrastructure = |
| | | |
| [[File:Nutrition Facts extraction flow.svg]] | | [[File:Nutrition Facts extraction flow.svg]] |
Line 41: |
Line 41: |
| | | |
| * Scoring des méthodes de détection de tableaux nutritionnels [https://drive.google.com/open?id=1-bo0v7eoRMtemZjcmqVzW0-0YMZaSOkB PDF] [https://drive.google.com/open?id=1KJ6H1HqHSHVBk3mWjEZZLLpcCt5whe6N docx] | | * Scoring des méthodes de détection de tableaux nutritionnels [https://drive.google.com/open?id=1-bo0v7eoRMtemZjcmqVzW0-0YMZaSOkB PDF] [https://drive.google.com/open?id=1KJ6H1HqHSHVBk3mWjEZZLLpcCt5whe6N docx] |
− | * Analyse approfondie sur la détection | + | * Analyse approfondie sur la détection de tableaux nutritionnels [https://drive.google.com/open?id=1nPxOGFgjKaKuUcnVC0nbNAijt1al9rvo pdf] |
− | de tableaux nutritionnels [https://drive.google.com/open?id=1nPxOGFgjKaKuUcnVC0nbNAijt1al9rvo pdf] | |
| | | |
| * Table extraction (V1): https://github.com/openfoodfacts/off-nutrition-table-extractor. Mainly focus on the nutritional table detection part, the extraction process remains quite simple. | | * Table extraction (V1): https://github.com/openfoodfacts/off-nutrition-table-extractor. Mainly focus on the nutritional table detection part, the extraction process remains quite simple. |
| * Table extraction (v2): https://github.com/cgandon/openfoodfacts-nutriments | | * Table extraction (v2): https://github.com/cgandon/openfoodfacts-nutriments |
| + | * Work by Sadok and Yichen as part of the Microsoft ShareAI program |
| + | ** TableNet |
| + | ** GraphNet |
| | | |
| = Other approaches = | | = Other approaches = |
Line 63: |
Line 65: |
| | | |
| * Test of an end-to-end model, through an API as a starter: Form Recognizer or Google Document AI API. If results are promising, we develop in-house end-to-end model, otherwise a layout model. | | * Test of an end-to-end model, through an API as a starter: Form Recognizer or Google Document AI API. If results are promising, we develop in-house end-to-end model, otherwise a layout model. |
− | * Thourough litterature review | + | * Thorough literature review |
| * Architecture choice | | * Architecture choice |
| * Annotation campaign | | * Annotation campaign |
Line 72: |
Line 74: |
| | | |
| Yichen: test of image preprocessing before Form Recognizer (layout model), test "manual" approaches. | | Yichen: test of image preprocessing before Form Recognizer (layout model), test "manual" approaches. |
| + | * tested a pre-processing method (ported from Mathematica to python) |
| + | * OCR results worsen after binarization |
| + | * testing method to morph deformed images (by detecting horizontal and vertical lines) |
| + | * "manual" approaches: compare positions of bounding boxes of OCR, use borders if we have some |
| + | ** manual horizontal or vertical scans of bounding boxes, using the angles of bounding boxes |
| + | ** robotoff has python classes for importing/analyzing Google Cloud Vision |
| | | |
− | Raphaël: Litterature review (to be continued) | + | Raphaël: Literature review (to be continued) |
| + | |
| + | = Notes 24/07 = |
| + | |
| + | * Ramzi: tested end-to-end model with learning |
| + | ** missing one step, results expected next week |
| + | * Yasmine + Yichen: working on pre-processing |
| + | ** Testing cylinders distortions -> flat rectangle |
| + | *** open cv |
| + | *** Tesseract performance very poor |
| + | |
| + | Next week: |
| + | * Ramzi |
| + | ** finish test end-to-end model |
| + | ** increase size of data set to get significant results |
| + | * Yasmine + Yichen |
| + | ** Cylinder unwrapping (classic + deep learning models) |
| + | *** review litterature |
| + | *** create data set for Ramzi using paid API "perfect label" |
| + | ** table structure detection : associate bounding boxes to key value pairs |
| + | * All |
| + | ** Create test set with cylindric photos + unwrapped photos |
| | | |
| = Resources = | | = Resources = |
Line 82: |
Line 111: |
| ** Step 2: Download Lobe - For Mac: https://aka.ms/DownloadLobeMac - For PC: https://aka.ms/DownloadLobeWindows | | ** Step 2: Download Lobe - For Mac: https://aka.ms/DownloadLobeMac - For PC: https://aka.ms/DownloadLobeWindows |
| * [https://flow.microsoft.com/en-us/ai-builder/ AI Builder] - import your photos right away into a Power App (or a Power Automate flow) so you can use the output in any real world application. | | * [https://flow.microsoft.com/en-us/ai-builder/ AI Builder] - import your photos right away into a Power App (or a Power Automate flow) so you can use the output in any real world application. |
| + | |
| + | = Quality metrics = |
| + | * [https://docs.google.com/document/d/1sJNZOq2Gs6gnNSCV2hPlnl4-3Z_SdQFQuorqoRw1cjA/edit Quality Metrics & Golden sets] |