Blog

Precision

Precision

In the previous blog post we defined the calorie proxy (or signal) that Pertinacity uses to simplify calorie counting:

  • Imagine squashing the food item.
  • Make a fist. Estimate how many fists would be the same size as the squashed food item.
  • Exclude drinks known to have zero calories (water, diet soda, tea, black coffee, etc.), but include other drinks like smoothies, alcohol, regular soda, milk, juice, etc.

Now let's evaluate the quality of that signal.

There are two parts to the evaluation: precision and accuracy. In colloquial language, these terms are often used interchangeably so I'll state my definitions here. In fact, since it's easier to work with spread (inverse precision) and bias (inverse accuracy), I'll define those terms:

spread: The standard deviation of errors in a measurement.  Lower spread means higher precision.
bias: The average value of errors in a measurement.  Lower bias measure higher accuracy.

If you ask, "Will I, on average, under- or over-report my daily calorie intake?" you're asking about accuracy. While using a calorie counting system, consistently under-reporting calories will prevent you from losing weight (or, at least, slow your rate of weight loss). Over-reporting calories would have the opposite effect.

If you ask, "For a fixed bias, how far over or under actual calorie intake can my estimate be from day to day?" you're asking about precision.

Put another way, bias tells you how well your measurements do on average and spread tells you how big the fluctuations are.

For example, if you over-report by 200 calories today and underreport by 200 calories tomorrow your measurements are unbiased (very accurate) and your spread is 200 calories (a measure of precision).

This post discusses an estimate of the precision of calorie intake measurements made using Pertinacity's calorie proxy. A future post will address accuracy.

Precision in Calorie Measurement

How precise does a measurement of calories intake need to be to be useful? The answer is not obvious, but we can make some progress by considering this:

If a calorie counting (measuring) system underestimates the calories you've consumed for the day by C calories while you're trying to reach a certain calorie target for that day, you'll be likely to overeat by C calories.

And vice-versa -- if you overestimate you'll undereat.

So greater imprecision in a measurement could induce greater fluctuations in calories consumed from day to day. That suggests one limit on our precision: We'd like to keep the fluctuations induced by our calorie counting system smaller than the fluctuations we'd experience with normal, ad libitum, eating.

This paper, The nature and individuality of within-subject variation in energy intake, examines data collected from 29 individuals over the course of one year (from the Beltsville one-year dietary intake study) and reports in Table 1 the mean and standard deviation of calories consumed for all participants. The average of [standard deviation]/mean for all participants is about 27% (median is 25%).

As another point of reference, we could ask how precise calorie counting is when using a table or database. The study, What are people really eating? The relation between energy intake derived from estimated diet records and intake determined to maintain body weight, reports (Table 1) a daily variation of the error in calorie estimation (calorie counting) to be 16% for males and 19% for females or 18% overall.

In summary, the size of the fluctuations of normal (ad libitum, not dieting) calorie intake are around 27%, and the error in calorie counting measurements using a calorie database is around 18%. We'll use these numbers below to help calculate and put into context the precision of Pertinacity.

Data

How precise is the signal used in Pertinacity? To gauge Pertinacity's precision we collected data comparing our signal to calorie database lookups using Amazon's Mechanical Turk (AMT). AMT is a system that lets Requestors enter data collection instructions -- Human Intelligence Tasks (HITs) -- for Workers to complete and submit.

Thirty-one (31) HIT requests were entered into AMT with these instructions:

Record what you eat for 24 hours

Report the time each food item was eaten
Give a short description of each food item
Report the number of calories in the food item by (a) using the nutrition label, (b) performing a web search, OR (c) using the calorie database website http://www.calorieking.com
Report the size (volume) of the food item by (i) imagining squashing the food item then (ii) making a fist and estimating how many fists would be the same size as the squashed food item.
INCLUDE drinks with calories such as alcohol, smoothies, soda, milk, juice, etc.
EXCLUDE drinks known to have zero calories like water, diet soda, tea, coffee, etc.
Example:
 7:00am, Bowl of oatmeal, 220 calories, 1 Fist
 7:00am, 8 oz glass of orange juice, 112 calories, 1 Fist 
 12:15pm, Big Mac, 550 calories, 2 Fists
 12:15pm, Small fries, 101 calories, 1 Fist 
 2:00, Popcorn, 65 calories, 1 Fist
 etc.

Note: Everyone's fist is a different size, so the fist estimates given in the above example should not be considered "correct answers".  They are merely reflective of the size of the requestor's fist.

Twenty-nine (29) responses were accepted. Two (2) were rejected because they reported fists but not calories. The HITs showed a median of 6 food items reported per day and a median of 2030 calories per day. The data were pooled across all HITs giving 187 food items.

Evaluation

The model

Calories/item = beta * [fists/item] + eps

was fit via linear regression. A proposed model with a constant term showed that the constant term was not statistically significant. (The physical interpretation is straightforward: If I were to eat nothing I'd report 0 calories and 0 fists.) The beta resulting from the regression was:

beta = [186.7 Calories/fist]

The term eps is the error in the estimation of calories for a single food item.

std(eps) = 183.0 Calories/item
mad(eps) = 87 Calories/item
kurtosis(eps) = 5.18

Where std(x) is the standard deviation of x, mad(x) is the median absolute value of x, kurtosis(x) is the excess kurtosis of x.

Also,

corr(fists/item, Calories/item) = 0.65

where corr(x,y) is the correlation between x and y.

A sampling of the pooled food items along with each item's estimation error, eps, is given here.

eps=286.65 calories=380 fists=0.5  4:00pm alfredo sauce (homemade) 380 calories 0.5 fist
eps=93.10 calories=1400 fists=7  4:00am 7 tacos 1400 calories 7 fists
eps=76.60 calories=450 fists=2  12:00 pm hamburger with lettuce ketchup and mayo 450 calories 2 fists
eps=66.65 calories=160 fists=.5  7:45pm tuna salad 160 calories 1/2 fist
eps=63.30 calories=250 fists=1  2:30pm 1/3lb ground beef 250 cal 1 fist
eps=63.30 calories=250 fists=1  4:30 pm ice cream bar 250 calories 1 fist
eps=59.90 calories=620 fists=3  11:00 am plate of spaghetti and meat sauce 620 calories 3 fists
eps=43.30 calories=230 fists=1  7:00pm 1 cup no bean chili 230 cal 1 fist
eps=26.60 calories=400 fists=2  9:30am 2 hot dogs 400 calories 2 fists
eps=26.60 calories=400 fists=2  3:00 pm chili con carne and chips 400 calories 2 fists
eps=23.30 calories=210 fists=1  4:32 pm 16 fl. oz. can of energy drink 210 calories 1 fist
eps=20.60 calories=394 fists=2  7am 2 pieces of sour dough toast and butter 394 calories 2 fists
eps=9.90 calories=570 fists=3  8:00pm lasagna 570 cal 3 fist
eps=6.60 calories=380 fists=2  12:45 p.m. - 1 medium mcdonald's fry 380 calories 2 fists
eps=-11.70 calories=175 fists=1  8:00 pm ice cream sandwich 175 calories 1 fist
eps=-53.40 calories=320 fists=2  5:00pm bowl of cereal 320 calories 2 fists
eps=-53.40 calories=320 fists=2  8:00am bowl of honey nut cereal 320 calories 2 fists
eps=-68.70 calories=118 fists=1  7:20 pm 1 fist homemade french fries. 118 calories.
eps=-86.70 calories=100 fists=1  1pm rice crackers 100 calories 15 crackers 1 fist
eps=-86.70 calories=100 fists=1  11:00am orange juice 100 calories 1 fist
eps=-100.05 calories=180 fists=1.5  5:15pm - 12 oz of chocolate milk 180 calories 1.5 fists
eps=-160.10 calories=400 fists=3  3:00 pm taco salad 400 calories 3 fists
eps=-373.40 calories=0 fists=2  7:37pm: water 2 8oz glasses 0 calories 2 fists

Notable features are: (1) calories tend to increase with fists, (2) the data set includes at least one misinterpretation of the instructions (see the last line; Water was reported violating the rule "EXCLUDE drinks known to have zero calories like water...".), (3) Some users desired half-fist precision. This information could be used to help improve the design of Pertinacity.

Note that since this data is pooled across HITs it is pooled across Workers so that some of the variation in the error term comes from (at least) differences between individuals' fist sizes, their interpretation of the instructions, and the set of foods they typically consume. When a single user uses Pertinacity this variation is not present so we consider the number std(eps)=183.0 Calories/item to be an upper bound.

We approximate the error in estimates of daily calorie intake with this rough calculation:

variation in daily error = sqrt(median(items/day)) * std(eps) = sqrt(6 items/day) * sqrt(day/item) * 183.0 Calories/item = 448 Calories/day
variation in daily error = sqrt(median(items/day)) * std(eps) / median(Calories/day) =
    sqrt(6 items/day) * sqrt(day/item) * 183.0 Calories/item / (2030 Calories/day) = 451 (Calories/day) / 2030 (Calories/day) = 22%

In short, based on this data, we could expect at most a variation of 22% in our estimate of "database calories" from "fists". Recall from above that database calories have an error to actual consumed calories that varies by about 18%. Assuming these two errors are uncorrelated we can estimate the variation in the error from fists to actual calories by

Pertinacity spread upper bound = sqrt(18%^2 + 22%^2) = 28%

This is the spread of the Pertinacity calories proxy defined near the top of this post. Again, because of the variation across Workers we treat this number as an upper bound. This number is just above the measurement of natural daily variation of calorie intake of 27% reported above. We conclude that this calorie proxy is near the threshold of suitability in that it should not induce fluctuations in a user's diet that are larger than they experience in normal eating.

Conclusion

Pertinacity uses a proxy for calories -- a signal -- rather than a database lookup. To understand how good the signal is we need to evaluate its precision and accuracy. In this blog post we estimate an upper limit on the spread (inverse precision) of the signal at 28% (compare to 18% for a calorie database lookup). We argue that a user who limited their eating using Pertinacity would experience fluctuations in daily calorie consumption similar to or less than those experienced in normal, ad libitum, eating.

Adherence via Simplicity

We saw in the previous blog post that calorie reduction works. You'll lose weight if you eat less -- i.e., if you eat few enough calories. The problem seems to be adherence. That is, people drift off their diets and begin eating too much and regain weight.

I stopped losing weight because I stopped counting calories and starting eating more. I stopped counting calories because it was too hard. It wasn't hard to do at first. It was even interesting to learn a little about the foods I was eating. When that novelty wore off however, I was faced with these difficulties:

  • It takes time to look foods up in a calorie database, even if that database is in a well-designed App.
  • It is difficult to know the ingredients and amounts of ingredients in restaurant food. (Ex., How many tbsp. of butter are on my bagel? What's in that delicious sauce on my fish?)
  • It is difficult to estimate the size of some foods (ex., Is that a medium apple or a large apple? How many ounces of chicken are on my plate?)
    • There can be many ingredients in home-cooked food that, ideally, should be measured individually and then looked up.

All of these things, for me, are quite tedious and distracting and result in fatigue and frustration. Even though the fatigue and frustration are mild, over time they're discouraging enough to make me quit.

So I hypothesize that I’ll persist in calorie counting/reduction if the process is simple enough.

Below I discuss the three simplifications made by Pertinacity in an attempt to improve adherence: the design of a calorie proxy, the design of a calorie limit, and the elimination of a weight-loss goal (and thus a "maintenance" mode).

Calorie Proxy

I want a method of measuring calorie intake that doesn't require a database search, works with any food (home-cooked, packaged, prepared), and works anywhere (home, work, vacation).

To simplify calorie intake measurement I estimate calories by proxy -- or "signal" in statistical terms. The signal is defined in the Pertinacity Instructions:

  • Imagine squashing the food item.
  • Make a fist. Estimate how many fists would be the same size as the squashed food item.
  • Exclude drinks known to have zero calories (water, diet soda, tea, black coffee, etc.), but include other drinks like smoothies, alcohol, regular soda, milk, juice, etc.

This solves the "any food" and "anywhere" problems. You measure an entree at a restaurant the same way you measure a bag of chips or a sandwich you make at home. Also, your "measuring device" -- your fist -- is always with you. You won't need to use a diet scale, measuring cups, etc. to get the job done. Using a fist to help estimate food volume is a technique borrowed from Portion Control).

[The next step is to verify that this proxy gives a good -- or, at least, good enough -- estimate of calories to be useful. After all, different foods have different calorie densities (calories / unit volume) so treating them all equally will result sometimes in overestimation and sometimes in underestimation. It seems fair to guess that these fist estimates won't be as precise as database lookups. How much less precise are they? How precise are database lookups in the first place? And what about accuracy? We'll explore these questions in future posts.]

Calorie Limit

Since we're estimating calorie intake by proxy we can't directly make use of a reference model like Dietary Reference Intakes . (Weight-loss systems often start with that number and compute a daily calorie limit by incorporating other information.) Instead we'll develop a direct proxy limit, i.e., a limit on the number of fist-sized portions we may eat today.

A simple way to do this is to ask, "How many fist-size portions do you usually eat?" Pertinacity assumes this is enough to maintain your weight and then asks you to eat a little less. The limit is determined by taking the average of the past two weeks of your daily counts of fist-sized portions (shown in the History Editor ) and subtracting a little bit.

The assumptions in computing a limit this way are:

  • You will eat foods today that are similar to foods you've been eating for the past two weeks.
  • You will engage in physical activity today that is similar to the physical activity you've been engaging in for the past two weeks.
  • Your basal metabolic rate today will be similar to what it has been over the past two weeks.

Since the average is taken over the past two weeks looking back from the current day, if you do start a new exercise program or change your diet dramatically Pertinacity will automatically adapt its recommendations as data accumulates. You just keep counting fist-sized portions as usual.

Goals and Maintenance

A typical first step in a diet program that uses calorie counting is to define a goal. The goal is specified as something like "Lose P pounds in W weeks." After you reach your goal you switch from a weight-losing diet to a maintenance diet.

How much weight can someone lose and in what period of time? I don't know.

Should you set a more achievable (less aggressive) goal so that you won't be discouraged? No, it doesn't seem to matter. One study, Are smaller weight losses or more achievable weight loss goals better in the long term for obese patients? examines this question and concludes "...these results do not support the hypothesis that obese patients should be encouraged to set lower weight-loss goals". Another study, Are Unrealistic Weight Loss Goals Associated with Outcomes for Overweight Women? concludes "Results suggest that lack of realism in weight loss goals is not important enough to justify counseling people to accept lower weight loss goals when trying to lose weight."

So if we don't know how quickly we can (or should) lose weight and it seems that progress in weight loss is the same for reasonable goals as for unreasonable goals, maybe we should just do away with goals. Maybe a goal isn't an important factor in designing a weight-loss strategy.

Doing away with the goal-setting stage affords us more simplification. It means that when you start using Pertinacity you don't have to figure out the answer to the question about a goal. It also means that there's no separate "maintenance mode". You do the same thing everyday forever. There's less to learn and less to do.

Conclusion

I hypothesized that if I could make calorie counting simple enough I'd keep at it -- i.e., I'd show greater adherence. By eliminating maintenance and an explicit weight-loss goal and by switching to a calorie proxy my weight-control strategy has become very simple: I estimate the number fist-sized portions I eat and tap '+' to count them with Pertinacity.

Is it effective? I'm very optimistic. As of this writing I've been using Pertinacity for a little over eight months and have steadily lost weight -- 12 pounds -- without weight regain and without fatigue or frustration.

Calorie Reduction and Calorie Counting

Calorie Reduction

Before thinking about a specific method for losing weight, I read about weight loss in general. My understanding is that (i) to lose weight you need to have a net energy imbalance (to burn more calories than you consume) and that (ii) “a calorie is a calorie”, i.e., it doesn’t directly matter (for the purpose of weight loss) which kinds of food you eat or limit.

Energy Imbalance

A nice review paper I found is A meta-analysis of the past 25 years of weight loss research using diet, exercise or diet plus exercise intervention. This paper defines dieting as an intervention which “include[s] some type of calorie restriction or reduced energy intake”. They conclude:

Weight loss research over the past 25 y has been very narrowly focused on a middle age population that is only moderately obese, while the interventions lasted for only short periods of time. The data shows, however, that a 15-week diet or diet plus exercise program, produces a weight loss of about 11 kg, with a 6.6 +/- 0.5 and 8.6 +/- 0.8 kg maintained loss after one year, respectively.

I interpret this as: Calorie reduction works to lose weight. Adding exercise makes it a little better. I chose to focus only on calorie intake reduction (rather than on exercise) because it produced the overwhelming majority of the effect in this meta-analysis and because the action required on my part is somewhat simpler: Eat less.

Calorie Type

Some popular diets recommend varying specifically carbohydrate, fat, or protein intake. This large study, Comparison of Weight-Loss Diets with Different Compositions of Fat, Protein, and Carbohydrates concludes “Reduced-calorie diets result in clinically meaningful weight loss regardless of which macronutrients they emphasize.”

My interpretation is that I shouldn’t bother controlling which kinds of calories I’m consuming. I should just focus on consuming less of them overall. That’s a helpful simplification.

Calorie Counting

A straightforward method of reducing your calorie consumption is to use a calorie counting system: Estimate the number of calories you’re eating each day and make sure you don’t eat too many. There are many books, apps, and websites that can help with this.

Calorie reduction results in weight loss. Calorie counting is a great way to measure your calorie intake so that you can control it and be sure you’re actually reducing it.

Short-Term

The problem for many people — including me — is that after losing weight for some period of time they gain it back. This paper, Meta-analysis: the effect of dietary counseling for weight loss finds that weight loss goes well for the first 3-12 months (see Figure 2), but then weight is regained:

…meta-regression suggest a change of approximately -0.1 BMI unit per month from 3 to 12 months of active programs and a regain of approximately 0.02 to 0.03 BMI unit per month during subsequent maintenance phases.

Results like this make me wonder if long-term weight loss is even possible. If not, maybe there's no point in trying.

Long-Term

Some evidence that it is possible to lose weight and maintain a healthy weight for the long-term comes from Long-term weight loss maintenance in the United States

More than one out of every six US adults who has ever been overweight or obese has accomplished LTWLM of at least 10%. This rate is significantly higher than those reported in clinical trials and many other observational studies, suggesting that US adults may be more successful at sustaining weight loss than previously thought.

The paper studies a sample of indivuduals designed to reflect the population of the US. There is also a discussion of factors that tend to help or harm long-term weight maintenance.

The National Weight Control Registry tracks 10,000 voluteers in an attempt to understand factors that influence long-term weight maintenance. This population is not sampled to reflect the general population (as in the previous paper), but the NWCR is very long-running and has a large amount of data. Some of this data is analyzed in Long-term weight loss maintenance. Of note, they review several studies and conclude

Thus, although the data are limited and the definitions varied across studies, it appears that ≈20% of overweight individuals are successful weight losers.

The paper goes on to analyze factors that might constribute to long-term weight maintenance.

These numbers -- "one out of every six" and "≈20%" -- are not large. That means most people fail to keep weight off for the long term.

Some succeed, though. I'm optimisitic enough to try to be one of them.

Adherence

Studies suggest that adherence to a diet plan is vital to its success:

  • The thesis Smart Phones and Dietary Tracking: A Feasibility Study reviews literature on this question:

    Making the process of recording food consumption and energy expenditure easier may encourage more people to develop this habit and maintain it for a greater length of time. Some research already points to this possibility. Arsand et al. (2008) studied the viability of using mobile phone technology to enhance accountability with diabetics and found that one of the key factors to success and sustainability was the mobility of the recording device (Arsand, Tufano, Ralston, & Hjortdahl, 2008) … The authors concluded that self- monitoring of diet and physical activity is predictive of weight control outcomes and suggested that future studies should focus on innovative ways to increase adherence to self-monitoring (Jelalian et al., 2010).

  • From Comparison of the Atkins, Ornish, Weight Watchers, and Zone Diets for Weight Loss and Heart Disease Risk Reduction:

    Dr. Dansinger advises dieters and physicians alike that it is not the diet itself that is most important; it is adherence to a diet that leads to weight loss and cardiac risk reduction. He encourages more research to identify practical techniques for increasing dietary adherence, including techniques to match individuals with the diet best suited to individual food preferences, lifestyle and medical conditions.

Conclusion

All of this suggests to me that

  1. It's possible to keep weight off for the long-term even if it's not the norm.
  2. I should use calorie counting to reduce my calorie consumption and ignore calorie type.
  3. Focusing on adherence -- getting myself to stick with it -- might be a way to keep weight off for the long term.

My hypothesis is that if a calorie counting system is simple enough I’ll keep using it, and if I do then I'll keep my calories under control and maintain a healthy weight. Ideally, I’d never stop using it and, as a result, never have an unhealthy weight.

Long-term Weight Control

Last year at this time I weighed 185 pounds. At my height of 6 feet, that made my BMI 25.1, just high enough to qualify for “Overweight”. I tried calorie counting for a few months and lost 10 pounds, but the effort and tedium of calorie counting — along with a plateau in weight loss — resulted in me being discouraged, counting calories only intermittently, and eventually regaining five pounds.

This was the third time in my life I’d lost and eventually regained weight, so I decided it was time to understand the problem better and find a solution that worked for me. Some of the questions I had were:

  • Is calorie counting a good way to lose weight?
  • Is it possible to keep weight off, or are we all just destined to gain it back eventually?
  • If so, how can I avoid regaining weight?

My working hypothesis is that if I can make calorie counting very simple, I’ll stick with it. Maybe if the level of effort is low enough I’ll keep going even when my focus is on family or work instead of losing weight, even when I’m tried or under stress, even when I feel discouraged by a plateau, and even after I feel like I’ve lost enough weight.

The results of this project are summarized in this slide deck, Long Term Weight Control, which was presented at the NYC Quantified Self Meetup. I gave that presentation while I was executing my calorie tracking strategy on a spreadsheet. I have since implemented the strategy in an iPhone App called Pertinacity which I use every day.

I am now 167 pounds with a BMI of 22.6 which puts me in the “Normal” BMI category.

I’ll expand on the presentation and the strategy in future blog posts. Please download the app and take a look at the instructions.