In the previous blog post we defined the calorie proxy (or signal) that Pertinacity uses to simplify calorie counting:
- Imagine squashing the food item.
- Make a fist. Estimate how many fists would be the same size as the squashed food item.
- Exclude drinks known to have zero calories (water, diet soda, tea, black coffee, etc.), but include other drinks like smoothies, alcohol, regular soda, milk, juice, etc.
Now let's evaluate the quality of that signal.
There are two parts to the evaluation: precision and accuracy. In colloquial language, these terms are often used interchangeably so I'll state my definitions here. In fact, since it's easier to work with spread (inverse precision) and bias (inverse accuracy), I'll define those terms:
spread: The standard deviation of errors in a measurement. Lower spread means higher precision. bias: The average value of errors in a measurement. Lower bias measure higher accuracy.
If you ask, "Will I, on average, under- or over-report my daily calorie intake?" you're asking about accuracy. While using a calorie counting system, consistently under-reporting calories will prevent you from losing weight (or, at least, slow your rate of weight loss). Over-reporting calories would have the opposite effect.
If you ask, "For a fixed bias, how far over or under actual calorie intake can my estimate be from day to day?" you're asking about precision.
Put another way, bias tells you how well your measurements do on average and spread tells you how big the fluctuations are.
For example, if you over-report by 200 calories today and underreport by 200 calories tomorrow your measurements are unbiased (very accurate) and your spread is 200 calories (a measure of precision).
This post discusses an estimate of the precision of calorie intake measurements made using Pertinacity's calorie proxy. A future post will address accuracy.
Precision in Calorie Measurement
How precise does a measurement of calories intake need to be to be useful? The answer is not obvious, but we can make some progress by considering this:
If a calorie counting (measuring) system underestimates the calories you've consumed for the day by C calories while you're trying to reach a certain calorie target for that day, you'll be likely to overeat by C calories.
And vice-versa -- if you overestimate you'll undereat.
So greater imprecision in a measurement could induce greater fluctuations in calories consumed from day to day. That suggests one limit on our precision: We'd like to keep the fluctuations induced by our calorie counting system smaller than the fluctuations we'd experience with normal, ad libitum, eating.
This paper, The nature and individuality of within-subject variation in energy intake, examines data collected from 29 individuals over the course of one year (from the Beltsville one-year dietary intake study) and reports in Table 1 the mean and standard deviation of calories consumed for all participants. The average of [standard deviation]/mean for all participants is about 27% (median is 25%).
As another point of reference, we could ask how precise calorie counting is when using a table or database. The study, What are people really eating? The relation between energy intake derived from estimated diet records and intake determined to maintain body weight, reports (Table 1) a daily variation of the error in calorie estimation (calorie counting) to be 16% for males and 19% for females or 18% overall.
In summary, the size of the fluctuations of normal (ad libitum, not dieting) calorie intake are around 27%, and the error in calorie counting measurements using a calorie database is around 18%. We'll use these numbers below to help calculate and put into context the precision of Pertinacity.
How precise is the signal used in Pertinacity? To gauge Pertinacity's precision we collected data comparing our signal to calorie database lookups using Amazon's Mechanical Turk (AMT). AMT is a system that lets Requestors enter data collection instructions -- Human Intelligence Tasks (HITs) -- for Workers to complete and submit.
Thirty-one (31) HIT requests were entered into AMT with these instructions:
Record what you eat for 24 hours Report the time each food item was eaten Give a short description of each food item Report the number of calories in the food item by (a) using the nutrition label, (b) performing a web search, OR (c) using the calorie database website http://www.calorieking.com Report the size (volume) of the food item by (i) imagining squashing the food item then (ii) making a fist and estimating how many fists would be the same size as the squashed food item. INCLUDE drinks with calories such as alcohol, smoothies, soda, milk, juice, etc. EXCLUDE drinks known to have zero calories like water, diet soda, tea, coffee, etc. Example: 7:00am, Bowl of oatmeal, 220 calories, 1 Fist 7:00am, 8 oz glass of orange juice, 112 calories, 1 Fist 12:15pm, Big Mac, 550 calories, 2 Fists 12:15pm, Small fries, 101 calories, 1 Fist 2:00, Popcorn, 65 calories, 1 Fist etc. Note: Everyone's fist is a different size, so the fist estimates given in the above example should not be considered "correct answers". They are merely reflective of the size of the requestor's fist.
Twenty-nine (29) responses were accepted. Two (2) were rejected because they reported fists but not calories. The HITs showed a median of 6 food items reported per day and a median of 2030 calories per day. The data were pooled across all HITs giving 187 food items.
Calories/item = beta * [fists/item] + eps
was fit via linear regression. A proposed model with a constant term showed that the constant term was not statistically significant. (The physical interpretation is straightforward: If I were to eat nothing I'd report 0 calories and 0 fists.) The beta resulting from the regression was:
beta = [186.7 Calories/fist]
The term eps is the error in the estimation of calories for a single food item.
std(eps) = 183.0 Calories/item mad(eps) = 87 Calories/item kurtosis(eps) = 5.18
Where std(x) is the standard deviation of x, mad(x) is the median absolute value of x, kurtosis(x) is the excess kurtosis of x.
corr(fists/item, Calories/item) = 0.65
where corr(x,y) is the correlation between x and y.
A sampling of the pooled food items along with each item's estimation error, eps, is given here.
eps=286.65 calories=380 fists=0.5 4:00pm alfredo sauce (homemade) 380 calories 0.5 fist eps=93.10 calories=1400 fists=7 4:00am 7 tacos 1400 calories 7 fists eps=76.60 calories=450 fists=2 12:00 pm hamburger with lettuce ketchup and mayo 450 calories 2 fists eps=66.65 calories=160 fists=.5 7:45pm tuna salad 160 calories 1/2 fist eps=63.30 calories=250 fists=1 2:30pm 1/3lb ground beef 250 cal 1 fist eps=63.30 calories=250 fists=1 4:30 pm ice cream bar 250 calories 1 fist eps=59.90 calories=620 fists=3 11:00 am plate of spaghetti and meat sauce 620 calories 3 fists eps=43.30 calories=230 fists=1 7:00pm 1 cup no bean chili 230 cal 1 fist eps=26.60 calories=400 fists=2 9:30am 2 hot dogs 400 calories 2 fists eps=26.60 calories=400 fists=2 3:00 pm chili con carne and chips 400 calories 2 fists eps=23.30 calories=210 fists=1 4:32 pm 16 fl. oz. can of energy drink 210 calories 1 fist eps=20.60 calories=394 fists=2 7am 2 pieces of sour dough toast and butter 394 calories 2 fists eps=9.90 calories=570 fists=3 8:00pm lasagna 570 cal 3 fist eps=6.60 calories=380 fists=2 12:45 p.m. - 1 medium mcdonald's fry 380 calories 2 fists eps=-11.70 calories=175 fists=1 8:00 pm ice cream sandwich 175 calories 1 fist eps=-53.40 calories=320 fists=2 5:00pm bowl of cereal 320 calories 2 fists eps=-53.40 calories=320 fists=2 8:00am bowl of honey nut cereal 320 calories 2 fists eps=-68.70 calories=118 fists=1 7:20 pm 1 fist homemade french fries. 118 calories. eps=-86.70 calories=100 fists=1 1pm rice crackers 100 calories 15 crackers 1 fist eps=-86.70 calories=100 fists=1 11:00am orange juice 100 calories 1 fist eps=-100.05 calories=180 fists=1.5 5:15pm - 12 oz of chocolate milk 180 calories 1.5 fists eps=-160.10 calories=400 fists=3 3:00 pm taco salad 400 calories 3 fists eps=-373.40 calories=0 fists=2 7:37pm: water 2 8oz glasses 0 calories 2 fists
Notable features are: (1) calories tend to increase with fists, (2) the data set includes at least one misinterpretation of the instructions (see the last line; Water was reported violating the rule "EXCLUDE drinks known to have zero calories like water...".), (3) Some users desired half-fist precision. This information could be used to help improve the design of Pertinacity.
Note that since this data is pooled across HITs it is pooled across Workers so that some of the variation in the error term comes from (at least) differences between individuals' fist sizes, their interpretation of the instructions, and the set of foods they typically consume. When a single user uses Pertinacity this variation is not present so we consider the number std(eps)=183.0 Calories/item to be an upper bound.
We approximate the error in estimates of daily calorie intake with this rough calculation:
variation in daily error = sqrt(median(items/day)) * std(eps) = sqrt(6 items/day) * sqrt(day/item) * 183.0 Calories/item = 448 Calories/day variation in daily error = sqrt(median(items/day)) * std(eps) / median(Calories/day) = sqrt(6 items/day) * sqrt(day/item) * 183.0 Calories/item / (2030 Calories/day) = 451 (Calories/day) / 2030 (Calories/day) = 22%
In short, based on this data, we could expect at most a variation of 22% in our estimate of "database calories" from "fists". Recall from above that database calories have an error to actual consumed calories that varies by about 18%. Assuming these two errors are uncorrelated we can estimate the variation in the error from fists to actual calories by
Pertinacity spread upper bound = sqrt(18%^2 + 22%^2) = 28%
This is the spread of the Pertinacity calories proxy defined near the top of this post. Again, because of the variation across Workers we treat this number as an upper bound. This number is just above the measurement of natural daily variation of calorie intake of 27% reported above. We conclude that this calorie proxy is near the threshold of suitability in that it should not induce fluctuations in a user's diet that are larger than they experience in normal eating.
Pertinacity uses a proxy for calories -- a signal -- rather than a database lookup. To understand how good the signal is we need to evaluate its precision and accuracy. In this blog post we estimate an upper limit on the spread (inverse precision) of the signal at 28% (compare to 18% for a calorie database lookup). We argue that a user who limited their eating using Pertinacity would experience fluctuations in daily calorie consumption similar to or less than those experienced in normal, ad libitum, eating.