Category/Nuts/Almonds/Journal

A journal has been kept of the cleanup process of this category in order to get a better feel what needs to be done, issues that arise and to assess the impact.

Befor starting there are 2943 products in the category Almonds.

The following steps were followed:

Definition

Almonds can be whole (I assume that they did not underwent any treatment). After they are shelled, the can be blanched (thin skin removed), salted or roasted. They can also be sliced, I assume the blanched ones.

Hunger game

Some usual errors: chocolate coated, spiced, as ingredient, part of a mix. There are also various ways to cut Almonds up: ground, chunks, etc. And there are various ways to roast them. 75 products have been added. Need to decide where to put all the spiced almond variants. And also the various spiced roasted almonds. After adding the first batch, Hunger Games makes more suggestions, so I lost count of how many have been added. Total is now 3210 products.

I went through all the Hunger Game Almond subcategories as well. I do not know whether these were already add to almonds, or whether they are new. I kept an eye on the associated info. So sometimes yes, sometimes no. This puts me in a doubt: should I say yes to an incorrect subcategory, in order to have the product listed under Almonds?

Categories

The Hunger games gives already some inspiration for subcategories:

• Unshelled almonds. not sure if there are any.
• Grilled almonds should be called Roasted almonds
• Almond peeled unpeeled or blanched and Shelled almonds seem the same, but have different Agribalyse
• If we have salted almonds, we need unsalted almonds;
• if we have roasted almonds we need unroasted almonds;
• And we can combine unsalted AND unroasted into raw almonds;

There are 336 associated categories. We expect only 10 or so. Time to cleanup and edit products.

Are Marcona almonds a thing?

There are some categories that seem to misuse Almonds:

• Almond drinks
• Chocolate covered almonds

Down to 54 categories. Stop here for the moment. Needed to edit 283 products.

Ingredients

At start there are 1482 ingredients associated with Almonds. Seems a lot of repair work,

Most errors are due to wrong classifications.

Down to 258 ingredients. Many ingredients are synonyms that are not yet in the taxonomy. Also often the product name is used as ingredient. I think I will remove those (or add in processing). By now added some 323 edits.

I no longer see the trees for the forest at 337 edits. What did I do and what not. Especially difficult to see for scripts like Thai.

Still 342 edits and 244 ingredients. Need to review again when all taxonomy changes have been implemented and the category reprocessed.

Languages

Well that was a lot of work. I edited 720 product names and/or categories: setting to the right language, moving to the correct category inside and outside the Almond category. Now 3393 products in the Almonds category.

Enough clean up done, so I can make the word clouds.

Quality

Only 13 data quality errors for some 100 products. In the end I edited 83 products.

What to do with the data of products that is clearly wrong? Maybe we should be more precise in how we handle and interpret data errors. I seem to distinguish two type of errors:

• formal errors - these are based on rules we have established (Energy not > 3700 kJ, etc);
• interpreted errors - these are due to our understanding of the formal errors and are therefore more precise: kJ/kcal inverted, polyols missing, etc;
• outlier errors - data points that outside a formal nutritional envelope of a (cleaned) category of products. We can subdivide these into values that are impossible (very large outliers) and strange (can not be correct);

Next question is what to do? Do we intervene, either by correcting or deleting? In any case we should inform the user what we did.

Graphing

Looking at the current values via graphs (energy versus x).

Salt

There are values up to 25 g, so a lot to repair. Normally salt should be well below 0.5%, except for salted almonds (up to 3%?). I could correct some 24 products. Some products have 10% salt, which seems very high. These could have made an error g<>mg. But we cannot be sure. When editing I check all values and repair if needed.

Proteins

Normal values are around 21 g. Very few too large, but many too small. Quite a lot where the per portion was not checked. 68 edits required.