Introduction

The Nutrition facts table data extraction project aims to automatically extract the nutrition facts values from photos of nutrition facts table.

Extracting nutritional facts from images would allow to fill values when nutritional facts are missing and to check values filled by users.

Why it's very important

We need nutrition facts to compute nutritional quality with the Nutri-Score
It takes 2 minutes per product to type in nutrition values, a process that is also tedious and error-prone
- 1M products x 2 minutes = 10 years of work full time without week-ends or vacations
Being able to add complete data for a product quickly is key to attain a critical mass of products in new countries

Two approaches have been tested so far:

Regexes used on the OCR output. A new /predict/nutrient endpoint has been added to Robotoff. This approach works best when nutritional information are not displayed in a table (“text” only).
A clustering approach that tries to estimate the number of columns and lines of the nutritional tables using K-means or DBSCAN algorithms. Not integrated to Robotoff.

de tableaux nutritionnels pdf

Detection of the table structure: identifying rows, columns and cells of the table. Most of the time rows corresponds to nutrition labels (protein, carbohydrate, energy,...) and columns to the quantity (100g, per portion, % of daily intake). By using pattern matching, we can label the column and rows and extract the values of the nutritional table for 100g or per portion.
Using an end-to-end model to directly extract nutritional values. This may be done by scoring candidates for each field (protein_100g, protein_serving, carbohydrate_100g,...), and selecting the highest scoring token as the field value. This is similar to Microsoft Form Recognizer API or Google Document API.

Each approach requires different kinds of annotation, so chosing the most effective approach at the beginning of the project is crucial.