Difference between revisions of "Nutritional values probability theory"

Latest revision as of 10:00, 10 October 2024

When analysing the nutritional values of groups as products, such as categories, one likes to know what the expected value and spread around that values of that group is. This might for instance help to detect subgroups and unusual products. Plotting the individual values of products in the group is already very useful, but one likes to do more.

The usual tools for determining an average and distribution is to assume that the probability distribution of the nutritional values follows a normal gaussian distribution. Using a gaussian distribution assumes that the values can get any value: from minus infinity to plus infinity. For nutritional values this is not the case. Nutritional values are fractions that go from 0 to an upper limit. For Energy this upper limit is 3700 kJ and for the nutrients this is 1. The Energy limit is reached if 100% of the nutrients consist of oil for instance.

That the normal distribution cannot be used is very well demonstrated by the Violin plots. Underlying the graph is the assumption of a normal distribution. This in turn leads to a plot that can go below zero.

If a normal distribution cannot be used, what should be used then? Some other probability distribution should be used.

Note: this page limits itself to statistical considerations for nutritional values. There is much material available on Internet, which describes things much better. The page page is more about what should and can be applied to nutritional values.

Probability distribution

There exist many probability distributions, so which one is applicable to nutritional values. What we are looking for lies in the category of continuous distributions, which means that the nutritional values are Real numbers. And the distribution should be on a bounded interval. The most general of these distributions is the Beta distribution.

Beta distribution

The formula for the beta distribution is give in the figure below.

In the formula the values x are between 0 and 1. The α and β determine the shape of the function. The Γ(z) is the Gamma function. The parameters α and β are positive integers (not 0).

Probably α and β will roughly equal for most nutritional values. If the nutritional value is close to 0, α=1 and β is large; if the nutritional value is close to 1, α is large and β=1.

The mean is define by α/(α+β) and the variance by αβ/[(αβ)²(α+β+1)].

Applicability

Is this distribution applicable to the nutritional values of the groups of products? And when is it not? And how would be able to detect that? If there are within a group two different subgroups that have clearly different nutritional values, then it is obvious this distribution can not be applied. And the subgroups should be analysed separately.

Scaling

The Beta distribution assumes that the x-values are between 0 and 1. The nutritional values published on products are either in grams per 100 gram or in grams per serving. All these numbers need to be scaled, so that they are between 0 and 1. The beta distribution can only be used to analyse a single nutrient per weight. The distribution encodes the other nutritions (and components) in the 1-x portion of the formula. This implies that the maximum of the distribution is always 1 and not some other value which needs to be estimated.

Analysis of the nutrients per energy is another discussion.

Multivariate considerations

The beta function described above assumes that the nutritional values are uniformly distributed between 0 and 1. This unfortunately is not always the case. If a product consists of one nutrient this is correct. But if there are more than one nutrient, only the sum of all nutrients is between 0 and 1. A single nutrient has than a maximum that is smaller than 1. But we do not know that maximum beforehand.

The published macronutrients do not represent 100% of the weight of a product. For instance water represents a large fraction by weight of a product, but its value is unknown. Even not all macronutrients are always available (fiber).

Dirichlet distribution

Thus there is an interdependence between the maxima of each nutrient, which is set by the total. A Dirichlet distribution would be more applicable.

Dirichlet Distribution

This Beta distribution is a special case of the Dirichlet distribution. The Dirichlet distribution is described by 5 axes (fat, carbohydrates, fiber, protein, salt). Each of the axes has its own parameter α_i.

The sum of all the α's gives the α_total. This is used the mean and variance for each axis, i.e. Mean_i= α_i/α_total. And the variance: α_i(α_total-α_i)/(α₀²(α_total-1)).

If we look only at a single nutrient, the corresponding Beta-distribution is defined by: Beta(α_i, α_tot-α_i), thus the β of the Beta distribution is α_tot-α_i.

Instead of nutrient it is better to talk about (weight) components. This implies that we miss one component: the one that is not a nutrient. Otherwise the total will not be 1 (100% weight).

Energy

The Dirichlet distribution can only be used for the nutrients, as the sum of the nutrients is 1 (100% of the product weight). Thus we cannot include the Energy in the formula. Something that is more generalised is required.

The first thing to look at is the Generalized Dirichlet which doubles the parameters for each axis. But is not clear if this would be helpful. No useful examples can be found. A critique of the Dirichlet distribution is the restricted covariance matrix.

Adding Energy to the set of variables to be analysed introducing another problem: the variables are no longer independent. There exist linear relations between some nutrients (fat, carbohydrates, fiber, proteins) and energy. Any multivariate distribution should take this in account.

Another possibility is to use a Copula. Have to look further into this to see whether it might be useful.

Sub-nutrients

For some macronutrients, there are also sub-nutrients. For instance for fat, saturated fat. And for carbohydrates the sugars. Could the play a role in a probability distribution? A keyword might be Nested Dirichlet distributions.

Estimates

The theory above only describes the formulas describing the distributions. In that case we know already all parameters α and β. We want to determine these α and β from actual data. This requires finding the best estimate for these parameters. There is no straightforward way (a formula) to go from the data to these values. It is required to solve some complex equations by numerical analysis.

Beta distribution two parameter case

In this case we assume that the data is distributed between 0 and 1. This implies that we are only looking for the α and β. The wikipedia page for the Beta distribution describes 3 approaches to finding these α and β. There are three methods to determine the α and β. The explanation is quite dense and not easy to transform to a practical approach.

For the Maximum Likelihood method a more easily accessible approach is available. In short this method tries to find the maximum of the distribution, i.e. where the second partial derivatives are zero.

The Excel file published on this site has been transformed into a Numbers spreadsheet. To use these spreadsheets you need to input the normalised data and define an initial α and β. At first it is not clear what to use as initial values. Some experimentation is required as not all initial values will converge to a solution. Using the solution for α and β we can now calculate mean and variance. And we can create a plot of the solution together with a distribution of the data:

Canned beetroot carbohydrates

Beta distribution three parameter case

Should an upper limit of 1 be used Or should this upper limit be estimated as well? We should implement an numerical estimation, but that requires doing some of the math first. The wikipedia page can serve as inspiration. There is no procedure to be found on Internet, so I made one up: Three parameter beta distribution estimation procedure.

Beta distribution examples

It necessary to investigate a bit more what the acceptable values for alpha and beta are for nutritional values. Examples of data are gathered on this page.

Upper limit values

Note that any value in a nutritional table that is indicated by an upper limit (i.e. <) should be replace by zero (0). However this might bias the actual distribution to 0. And if we use the limits as actual values we might bias the actual distribution to a too high level.

@@ Line 6: / Line 6: @@
 If a normal distribution cannot be used, what should be used then? Some other [https://en.wikipedia.org/wiki/Probability_distribution probability distribution] should be used.
+Note: this page limits itself to statistical considerations for nutritional values. There is much material available on Internet, which describes things much better. The page page is more about what should and can be applied to nutritional values.
 == Probability distribution ==
 There exist many [https://en.wikipedia.org/wiki/List_of_probability_distributions probability distributions], so which one is applicable to nutritional values. What we are looking for lies in the category of continuous distributions, which means that the nutritional values are Real numbers. And the distribution should be on a [https://en.wikipedia.org/wiki/List_of_probability_distributions#Supported_on_a_bounded_interval bounded interval]. The most general of these distributions is the [https://en.wikipedia.org/wiki/Beta_distribution Beta distribution].
 === Beta distribution ===
 The formula for the beta distribution is give in the figure below.
 [[File:BetaProbabilityDistribution.png|300px|thumb|center]]
-In the formula the values x are between 0 and 1. The α and β determine the shape of the function. The Γ(z) is the Gamma function, i.e (z-1)! The parameters α and β are positive integers (not 0).
+In the formula the values x are between 0 and 1. The α and β determine the shape of the function. The Γ(z) is the [https://en.wikipedia.org/wiki/Gamma_function Gamma function]. The parameters α and β are positive integers (not 0).
 Probably α and β will roughly equal for most nutritional values. If the nutritional value is close to 0, α=1 and β is large; if the nutritional value is close to 1, α is large and β=1.
 The mean is define by α/(α+β) and the variance by αβ/[(αβ)<sup>2</sup>(α+β+1)].
+=== Applicability ===
+Is this distribution applicable to the nutritional values of the groups of products? And when is it not? And how would be able to detect that? If there are within a group two different subgroups that have clearly different nutritional values, then it is obvious this distribution can not be applied. And the subgroups should be analysed separately.
 === Scaling ===
-The Beta distribution assumes that the x is between 0 and 1. For the Energy this is not the case as the maximum allowed value is 3700 kJ. One has to scale the x values, so that they lie between 0 and 1.
+The Beta distribution assumes that the x-values are between 0 and 1. The nutritional values published on products are either in grams per 100 gram or in grams per serving. All these numbers need to be scaled, so that they are between 0 and 1. The beta distribution can only be used to analyse a single nutrient per weight. The distribution encodes the other nutritions (and components) in the 1-x portion of the formula. This implies that the maximum of the distribution is always 1 and not some other value which needs to be estimated.
+Analysis of the nutrients per energy is another discussion.
 === Multivariate considerations ===
-The beta function described above assumes that the nutritional values are normally distributed between 0 and 1. This unfortunately is not always the case. I a product consists of one nutrient this is correct. But if there are more than one nutrient, only the sum of all nutrients is between 0 and 1. A single nutrient has than a maximum that is smaller than 1. But we do not know the maximum beforehand.
+The beta function described above assumes that the nutritional values are uniformly distributed between 0 and 1. This unfortunately is not always the case. If a product consists of one nutrient this is correct. But if there are more than one nutrient, only the sum of all nutrients is between 0 and 1. A single nutrient has than a maximum that is smaller than 1. But we do not know that maximum beforehand.
+The published macronutrients do not represent 100% of the weight of a product. For instance water represents a large fraction by weight of a product, but its value is unknown. Even not all macronutrients are always available (fiber).
-Should the maximum of the published macronutrients be 100? Do the published macronutrients (fat, carbohydrates, fiber, proteins, salt) add up to 100%, or are the also other macronutrients that should have been included?  If no fiber values is published the sum will certainly be less than 100%.
+==== Dirichlet distribution ====
+Thus there is an interdependence between the maxima of each nutrient, which is set by the total. A [https://en.wikipedia.org/wiki/Dirichlet_distribution Dirichlet distribution] would be more applicable.
+[[File:DirichletDistribution.png|300 px|thumb|right|Dirichlet Distribution]]
+This Beta distribution is a special case of the Dirichlet distribution. The Dirichlet distribution is described by 5 axes (fat, carbohydrates, fiber, protein, salt). Each of the axes has its own parameter α<sub>i</sub>.
-Thus there is an interdependence between the maxima of each nutrient, which is set by the total. Some kind of [https://en.wikipedia.org/wiki/Dirichlet_distribution multivariate beta distribution] would be more in place.
+The sum of all the α's gives the α<sub>total</sub>. This is used the mean and variance for each axis, i.e. Mean<sub>i</sub>= α<sub>i</sub>/α<sub>total</sub>. And the variance: α<sub>i</sub>(α<sub>total</sub>-α<sub>i</sub>)/(α<sub>0</sub><sup>2</sup>(α<sub>total</sub>-1)).
-=== Adding energy ===
+If we look only at a single nutrient, the corresponding Beta-distribution is defined by: Beta(α<sub>i</sub>, α<sub>tot</sub>-α<sub>i</sub>), thus the β of the Beta distribution is α<sub>tot</sub>-α<sub>i</sub>.
-How could a probability distribution include the energy component? The energy is expressed as a function between the nutrients, thus adding an additional constraint.
+Instead of nutrient it is better to talk about (weight) components. This implies that we miss one component: the one that is not a nutrient. Otherwise the total will not be 1 (100% weight).
+=== Energy ===
+The Dirichlet distribution can only be used for the nutrients, as the sum of the nutrients is 1 (100% of the product weight). Thus we cannot include the Energy in the formula. Something that is more generalised is required.
+The first thing to look at is the [https://en.wikipedia.org/wiki/Generalized_Dirichlet_distribution Generalized Dirichlet] which doubles the parameters for each axis. But is not clear if this would be helpful. No useful examples can be found. A critique of the Dirichlet distribution is the restricted covariance matrix.
+Adding Energy to the set of variables to be analysed introducing another problem: the variables are no longer independent. There exist linear relations between some nutrients (fat, carbohydrates, fiber, proteins) and energy. Any multivariate distribution should take this in account.
+Another possibility is to use a [https://en.wikipedia.org/wiki/Copula_(statistics) Copula]. Have to look further into this to see whether it might be useful.
 === Sub-nutrients ===
-For some macronutrients, there are also sub-nutrients. For instance for fat, saturated fat. And for carbohydrates the sugars. Could the play a role in a probability distribution?
+For some macronutrients, there are also sub-nutrients. For instance for fat, saturated fat. And for carbohydrates the sugars. Could the play a role in a probability distribution? A keyword might be '''Nested Dirichlet distributions'''.
-Finding the correct probability distribution sounds like a student project.
+== Estimates ==
+The theory above only describes the formulas describing the distributions. In that case we know already all parameters α and β. We want to determine these α and β from actual data. This requires finding the best estimate for these parameters. There is no straightforward way (a formula) to go from the data to these values. It is required to solve some complex equations by numerical analysis.
-== Estimates ==
+=== Beta distribution two parameter case ===
-The nutritional values for a group of products can be used to estimate the values of the beta function. From [https://en.wikipedia.org/wiki/Beta_distribution#Two_unknown_parameters wikipedia]:
+In this case we assume that the data is distributed between 0 and 1. This implies that we are only looking for the α and β. The wikipedia page for the Beta distribution describes 3 approaches to finding these α and β. There are three methods to determine the α and β. The  [https://en.wikipedia.org/wiki/Beta_distribution#Two_unknown_parameters explanation] is quite dense and not easy to transform to a practical approach.
-[[File:BetaProbabilityDistributionEstimates.png|500px|thumb|center]]
+For the Maximum Likelihood method a more easily accessible approach is [https://real-statistics.com/distribution-fitting/distribution-fitting-via-maximum-likelihood/fitting-beta-distribution-parameters-mle/ available]. In short this method tries to find the maximum of the distribution, i.e. where the second partial derivatives are zero.
-It is also possible to estimate the maximum of the distribution, which we might need if we want to do things correct. The wikipedia page gives an extensive explanation. This analysis needs to be redone as we need to constrain any value to the lower limit of zero.
+The Excel file published on this [https://real-statistics.com/distribution-fitting/distribution-fitting-via-maximum-likelihood/fitting-beta-distribution-parameters-mle site] has been transformed into a [https://apps.apple.com/gb/app/numbers/id361304891 Numbers] spreadsheet. To use these spreadsheets you need to input the normalised data and define an initial α and β. At first it is not clear what to use as initial values. Some experimentation is required as not all initial values will converge to a solution. Using the solution for α and β we can now calculate mean and variance. And we can create a plot of the solution together with a distribution of the data:
+[[File:BeetrootCarbohydratesBeta.png|300px|thumb|center|Canned beetroot carbohydrates]]
-The above functions do not work in practice as the observed distributions have a variance that is too large.
+=== Beta distribution three parameter case ===
+Should an upper limit of 1 be used Or should this upper limit be estimated as well? We should implement an numerical estimation, but that requires doing some of the math first. The wikipedia page can serve as inspiration. There is no procedure to be found on Internet, so I made one up: [[Three parameter beta distribution estimation procedure]].
-=== Violin plots ===
+=== Beta distribution examples ===
-For violin plots an adapted [https://en.wikipedia.org/wiki/Kernel_density_estimation Kernel Density Function] must be used, which uses the beta probability distribution. The advice is to cut the kernel functions at the bounds 0 and 1.
+It necessary to investigate a bit more what the acceptable values for alpha and beta are for nutritional values. Examples of data are gathered on this [[Nutritional values probability examples|page]].
 === Upper limit values ===
-Note that any value in a nutritional table that is indicated by an upper limit (i.e. <) should be replace by zero (0).
+Note that any value in a nutritional table that is indicated by an upper limit (i.e. <) should be replace by zero (0). However this might bias the actual distribution to 0. And if we use the limits as actual values we might bias the actual distribution to a too high level.