Line 1: |
Line 1: |
| + | [[Category:Robotoff]] |
| == Finding opportunities == | | == Finding opportunities == |
| + | === Option 1 === |
| # Download the OCR JSONL | | # Download the OCR JSONL |
− | # <pre>grep -o '\<WORD\>' | wc -l</pre> | + | # <pre>zgrep -n '05.01.07' FILE | wc -l</pre> |
− | == Using REGEXes == | + | |
| + | === Option 2 === |
| + | # Right click on an image in a product page, in edit mode (eg 1.jpg) |
| + | # Replace `.jpg` by `.json` |
| + | # Find patterns or `logoAnnotations` in the JSON |
| + | |
| + | == Adding detections using REGEXes == |
| https://github.com/openfoodfacts/robotoff/tree/master/robotoff/insights/ocr | | https://github.com/openfoodfacts/robotoff/tree/master/robotoff/insights/ocr |
| | | |
| + | === Add a function to add the desired match === |
| + | |
| + | * Choose the relevant file within the insights/ocr directory (in this case: https://github.com/openfoodfacts/robotoff/blob/master/robotoff/insights/ocr/packager_code.py) |
| + | |
| + | * Write the function |
| + | <pre> |
| + | def process_fsc_match(match) -> str: |
| + | fsc_code = match.group(1) |
| + | return "FSC-{}".format(fsc_code).upper() |
| + | </pre> |
| + | |
| + | === Add the function to the processing pipeline to plug it to Robotoff === |
| + | This happens in the same file. |
| + | <pre> |
| + | PACKAGER_CODE: Dict[str, OCRRegex] = { |
| + | "fr_emb": OCRRegex( |
| + | re.compile(r"emb ?(\d ?\d ?\d ?\d ?\d) ?([a-z])?(?![a-z0-9])"), |
| + | field=OCRField.text_annotations, |
| + | lowercase=True, |
| + | processing_func=process_fr_emb_match, |
| + | ), |
| + | "fsc": OCRRegex( |
| + | re.compile(r"fsc.? ?(c\d{6})"), |
| + | field=OCRField.text_annotations, |
| + | lowercase=True, |
| + | processing_func=process_fsc_match, |
| + | ), |
| + | "eu_fr": OCRRegex( |
| + | re.compile( |
| + | |
| + | </pre> |
| | | |
− | == Using Flashtext == | + | === Grab yourself a sandwich === |
| + | |
| + | == Adding detections using Flashtext == |
| https://github.com/openfoodfacts/robotoff/tree/master/data/ocr | | https://github.com/openfoodfacts/robotoff/tree/master/data/ocr |
| + | === Expanding existing detections === |
| + | * Choose a file matching the detection you want to add |
| + | In this example: https://github.com/openfoodfacts/robotoff/blob/master/data/ocr/brand_logo_annotation.txt |
| + | * Add entries one per line. The raw `logoAnnotation` you find in the OCR JSON is on the left, the value you want to add in the brand field is on the right. |
| + | <pre> |
| + | alnatura||Alnatura |
| + | Andros France||Andros |
| + | andros||Andros |
| + | </pre> |
| + | |
| + | === Add a new type of detection === |
| + | * Create a file `fishing_flashtext.txt` and add one value to extract per line |
| + | |
| + | === Add it to the Robotoff settings === |
| + | <pre> |
| + | robotoff/settings.py |
| + | OCR_LABEL_WHITELIST_DATA_PATH = OCR_DATA_DIR / "label_whitelist.txt" |
| + | OCR_FISHING_FLASHTEXT_DATA_PATH = OCR_DATA_DIR / "fishing_flashtext.txt" |
| + | </pre> |
| + | |
| + | === Plug it into the OCR pipeline === |
| + | <pre> |
| + | robotoff/insights/ocr/packager_code.py |
| + | codes = text_file_iter(settings.OCR_FISHING_FLASHTEXT_DATA_PATH) |
| + | return generate_keyword_processor(("{}||{}".format(c.upper(), c) for c in codes)) |
| + | </pre> |