Recipe/Estimator/Solver Evaluation

The Recipe Estimator allows a comparison between different solvers. This page evaluates the different in order to determine which solver is best for which situation.

Quality of fit

The quality of a fit can be judged by looking at the variance, the sum of the squared differences between fit and product for each (macro-)nutrient. Roughly the amount of variance less than 1 is a excellent fit, above 5 (?) is an estimation failure.

Estimation failures

The estimation does not work in multiple situations. Several situations have been identified and are discussed on this page. Products where the estimation fails will not be used in the solver evaluation.

Solvers

The Recipe Estimator supports multiple solvers:

• Differential Evolution
• GLOP Linear solver
• NNLS (constrained) - the order of the ingredients is enforced;
• NNLS (unconstrained) - the ingredient order is not enforced;
• Simple inverse power
• Simplified product opener

Weighting

The solvers can be used with weighted and unweighted data. The weights will reinforce some nutrients more than others.

Compound ingredients

Some products present an ingredient list with compound ingredients, like:

almond milk 63% (water, almond puree 15%), rapeseed oil, rice flour, water,

The almond milk is here a compound ingredient consisting of 2 ingredients water and almond puree

Most solvers need special code to support these kind of structures.

Comparing Solver results

The data is taken from a test set and published here . Multiple questions can be posed and should be tested.

One Ingredient

Ciqual usability

Can the Ciqual data to be used to fit the data. In order to test this we can look at single ingredient products and check how well Ciqual fits to actual products. It is expected that Ciqual is an average value for a product. Individual products will have different value, which would reflect the products statistical variance.

The graph shows the distribution of the NNLS variance for 37 products with 1 ingredient. Some 19 products have an acceptable variance (<2). The other variance vary wildly. Is this expected or does it say something about the data quality of producers?

GLOP versus NNLS (constrained)

Products with a single ingredient allows us to see which solver gives better results, i.e. with a lower variance. We would expect to get the same results for the various solvers. In the literature checks are reported that this is indeed the case.

The histogram shows the variance ratio's (GLOP/NNLS) for 43 single ingredient products. The interval 0-1 shows the products where the GLOP solver is better. Above 1 the NNLS solver is better.

In short an unexpected result.

Weighted versus unweighted

The question here is to ascertain whether weighting is useful. Not sure what the expected effect would be for single ingredient products, neither what the expected impact on the variance would be. Maybe the weighted focus on a single nutrient will that nutrient better, which in turn results in a better (or worse) fit.

For the investigated products (N=37) the weight variance is larger than the unweighted variance. The weighted variance is based on the recipe solution shown without any weights used in the variance calculation or have weights been added? What is the recipe solution of the unweighted variance.

In any case no evidence that the weighted helps.

Two ingredients

In this analysis only products that have 2 ingredients with identified Ciqual equivalents have been used.

Ciqual usability

The question is how well the Ciqual ingredients can explain the nutritional valus of the product. The unweighted NNLS variances are used.

Products with a variance below 1 can be seen as a perfect fit (10 out of 29). Other products can have a very large variance. Need to check out whether this can be due to the variability of the ingredients.

GLOP versus NNLS (constrained)

It can be confirmed with these products as well that NNLS performs better than GLOP (why?)

Three ingredients

For 20 products checked, in no case GLOP or weighted worked better. The distribution of variance has less extreme values.

Four ingredients

No new insights after look at 22 products.

Five ingredients

No new insights after look at 22 products.

Six ingredients

How many products have really 6 ingredients that contribute to the nutritional values? The sixth ingredient is often flavouring, thickener, etc.

Seven ingredients

No new conclusions (N=17). Note that we have a set of 7 equations with 7 unknowns

Eight ingredients

With more ingredients the chance to find ingredients that are unknown or do not contribute to the nutrition only increases.

Acceptability

In general the fitting results are pretty good. But what is pretty good? When should a fit be accepted and when not? What is the largest error in a recipe that could be accepted. How does the natural variability of ingredients impact the variance? And when is this variance due to an error in the nutritional values?

By looking at the variance distribution of a large set of products with different ingredients, it is possible to define a variance cutoff: above a this cutoff value the fit is rejected. Probably there is an expected theoretical variance distribution given the errors in the ingredients. Ideally the cutoff is defined per product category, which has the same ingredients and thus the same variability. That is for later. At first a cutoff for any product can be defined with any ingredients. This will then be based on ingredients with the largest variation.

Two ingredient products

Calculating more LS variances for products with 2 ingredients in the France known percentages set, allows to create and fit a distribution. Several variance were double, suspecting that the producer just copied the data. These points were left out. This gives this graph:

With this we can define a threshold of 200 (90% percentile). Above this value the fit cannot be trusted and the product should be checked. A 90% percentile might seem a bit low, but is due to the correlation of the nutritional values of ingredients. A threshold of 200 implies that in the worst case a single nutritional might be off by 14 (lower or higher); or 2 nutritional values off by 10.

As already noted before: ideally this analysis should be done per product category. Nuts seem to have a much larger variability than tomatoes. Using such a large threshold would imply than we accept tomato product fits that are actually wrong.

The plot above shows what happens when we enlarge the dataset, but removed products in the en:nuts category. The mean RSS falls from 92.7 to 69.1, which is an indication of the error in the fit. This in turn says something about the variability of the ingredients, like sesame.

Three ingredient products

The same analysis has been done for products with three ingredients.

What stands out is that this graph is much more like an expected chi-squared distribution. The cutoff has become much lower. Mean RSS is now down to 17, which means a maximum recipe error of only 4 (on 100g).

Visual inspection of the products with large fit errors, shows that also the nuts form a problem, as well as some other ingredients (maize). A fit on the cleaned set reduces the RSS to 14 and the cutoff to 32.

Four ingredient products

The same for four ingredients:

The fits are even better for this dataset with a mean RSS of 4.48 and a P90 threshold of 8.23. The more ingredients added the better the fit becomes.

The small errors are a bit of a worry. Wouldn't you expect a larger error that reflects the variability of the ingredients? Shouldn't the error distribution stay similar as single and double ingredient products?

Five ingredient products

Some observations:

• compounds - the compound ingredients start to appear. NNLS does not support that at the moment. The GLOP and Differential Evolution give better results. These products have been excluded from the analysis above;
• wrong Ciqual - in this 5 ingredient set the chocolate appear, which contain cocoa mass, which is not in Ciqual;

Issues

Sometime the Recipe Estimator does not work well. There can be multiple issues for this: some can be solved. The issues listed below:

• Missing ingredients - from the list of ingredients it is clear that something is wrong. Checking the product will confirm this. Solution: repair the product (example, example 2);
• Ciqual equivalent wrong - an equivalent Ciqual ingredient has been selected, which will probably result in a wrong ingredient list (example);
• Interchangeable ingredients - a product can have multiple ingredients with the same nutritional profile. The algorithms cannot determine, which one is more prevalent. Constrained NNLS assigns each ingredient 50%, GLOP does a 66%/34% split (example; example2). The NNLS unconstrained fit will reduce the recipe to a single ingredient;
• Ciqual equivalent not identified (solvable) - one of the ingredients is marked as Unknown. Selecting another Ciqual from the list solves the issue. This can be a parsing issue (example - d'Occitanie should be stripped off; example; example; example);
• Ciqual equivalent not identified (unsolvable) - not all ingredients are listed in Ciqual and thus can not be identified (example probably not an issue on the recipe; example - what is camu-camu?);
• Ingredients parsing issue - the ingredients have not well been extracted (example - ou (or) not taken into account; example);
• Ingredients parsing ending - the end of the ingredients has an issue (example - some text must be deleted);
• Fermented products - the nutritional values of fermented products (cheeses) do not reflect the ingredients as a transformation of sugars happened during processing (example);
• Missing nutritional info - these can not be handles by the recipe estimator (example; example);
• Duplicates - sometimes different products have exactly the same fit. Probably the producer just copied the nutritional values. Example (3336971510024 and 3336971508106);
• Under specified ingredients - some ingredients exist in more specific forms. Using these specific forms give better fit results. example (sunflower oil), wheat flour;

Remarkable Products

Sometimes there is a product where the solver fails for some unclear reason. These can be investigated further.

REGAÑA

For this product the GLOP solver gives the best results. The stepper however makes it possible to find an even better solution.

Redoing the NNLS-fit, results in a variance of 14.9.

The plot shows from which ingredients the nutritional values come: the deviation seem mainly due to the wheat of Ciqual (to much fiber, not enough proteins).

Why did the NNLS of the Recipe Estimator fail?

Rösti

For this product it helps to use the steppers to improve the results. The solvers did fail.

Redoing the fit with R gives a variance of 1.26.

The blame plot shows deviations for mainly sugar and protein. As these are mainly due to the potatoes, another variety should be used.

There is no reason the Recipe Estimator fits failed.

Croustillant 3 noix

The NNLS of the Recipe estimator gives worse results than GLOP. What happened?

The amount of fiber is on the low side. Probably due to the type of wheat flour used.

Ciqual

Missing entries

Some ciqual entries are missing from the taxonomy or proxies can be used. The changes have been added to the ingredients taxonomy and a pull request has been created. Check the request to see the details:

• 13671

Which ciqual?

For some products it is not obvious which ingredient should be used:

• Golden Allumettes (3256830000582) uses potatoes.
  • boiled gives a variance of 0.41,
  • raw of 0.62;
• Mais Waffeln mit Meersalz Bio (20884697) uses corn.
  • raw corn - variance 1675;
  • sweet corn, canned - variance 975;
  • corn frozen raw - variance 295;
• BIO-KNÄCKEBROT ROGGEN (20117566) uses: Roggenvollkornmehl
  • Rye, wholegrain, raw (9390) - variance 3.81;
  • Rye flour, type 170 (9530) - variance 421.7;
• Gomasio (3350033828346) use some sesame seed (complet). The husked gives better results, contrary to what is expected. Producer issue?):
  • Sesame seed, husked (15035) - variance 80.5;
  • Sesame seed (15010) - variance 140.78;
• Raisins secs sultanines (3760324844045) raisin or grapes:
  • Raisin (13046) - variance 0,01;
  • Grape, raw (13112) - variance 2,27;

Conclusions

• NNLS (constrained) is the preferred algorithm, it results in small variances than any of the other algorithms. This is no surprise as the constrained NNLS is in reality just a straightforward LS, which is solvable mathematically (no numerical approaches are needed). There is a single solution to this problem;
• Solvers cannot distinguish between ingredients with the same nutritional profile (sugar is the same as honey, which is the same as agave sirup for example);
• The list of Ciqual data now also contains compound ingredients. These could be removed, which would reduce the possible errors. It is not a big issue as the matching works very well.
• The matching problems with Ciqual ingredients that should have worked, seem to be solvable by extending the taxonomies (ingredients and ingredients processing);
• Having a test set to check out algorithms, etc is a good idea. The current test set has however to many products with issues. A clean one with 100% correct products should be defined;
• Some categories, like herbal teas, have essentially no nutritional information. These could be skipped in the recipe estimation;
• Cutoff thresholds can be defined for variances. Variances above the threshold indicate there is an issue with the fit and detailed inspection is required. Variances can be defined for specific recipe subsets (#ingredients) and categories;
• Setup a Ciqual chain to investigate if there are other Ciqual ingredients that result in a better fit;