Blog

Correlations

Correlation tells us how well two measurements of the same thing compare to each other. Correlation has two nice properties:

(1) It is a simple number without dimension (ex., feet or seconds or calories).  This means we can use it to compare different types of measurements.
(2) Its magnitude falls between 0 and 1.  This means there is a clear best value and a clear worst value.

What's not so clear is what the in-between values mean. How good is good enough? That depends on the application. To get some intuition for correlation we can look at several examples. First, let me explain where these examples come from.

Cheaper Measurements

One way to evaluate the precision of a measurement is to compute the correlation of the device's measurement with a known, "true" value. Since genuinely "true" values are unknowable, usually a measurement from a more precise -- but usually also more expensive -- reference measuring device stands in for the true value.

The interest in evaluating measurements this way usually stems from the desire to take a certain kind of measurement more cheaply. (I'm using "cheaply" in a generic sense: less complex, less difficult, or just costing less money.)

For example, the motivation for studying the Body Mass Index (BMI) -- a proxy for body fat -- was that BMI is very easily computed from height and weight. Alternative measures of body fat can be labor intensive when applied to many people (ex., skin calipers) or even when applied to one person (ex., weighing a person under water).

To evaluate the quality of BMI, one could compare BMI for many individuals to one of the more expensive methods of measuring body fat. If BMI were to correlate well with the more expensive measurement, you could then rely on BMI alone for future measurements. This was the approach taken by Indices of relative weight and obesity in 1972. BMI is still widely used today.

In the blog posts Precision and Dynamics we took the same approach with Pertinacity's fist-sized portion proxy for calories and Pertinacity's method of setting a limit for proxy calories. In these measurements we used as the "true", or reference, values the calorie values found in a calorie database.

Examples

Below are some examples of published correlation values for familiar measurement devices and indicators along with Pertinacity's correlations.

Measurement, Reference MeasurementCorrelation
Breathalyzer (Cooperative Patients), Serum Blood Alcohol Level
Gibb KA, Yee AS, Johnston CC, Martin SD, Nowak RM. Accuracy and usefulness of a breath alcohol analyzer. Ann Emerg Med. 1984;13(7):516-20.
.96
Ear Thermometer, Pulmonary Artery Temperature
Erickson RS, Meyer LT. Accuracy of infrared ear thermometry and other temperature methods in adul ts. Am J Crit Care. 1994;3(1):40-54.
~.89 (.87-.91)
Jawbone UP Steps, Handcounted Steps
Stackpool, Caitlin M. Accuracy of various activity trackers in estimating steps taken and energy expenditure. 2013
~.83 (.34, .98, .99, .99)
BMI, Water-Weighing for Density
Keys A, Fidanza F, Karvonen MJ, Kimura N, Taylor HL. Indices of relative weight and obesity. J Chronic Dis. 1972;25(6):329-43.
~.76 (.66, .85)
BMI, Skin Calipers
Keys A, Fidanza F, Karvonen MJ, Kimura N, Taylor HL. Indices of relative weight and obesity. J Chronic Dis. 1972;25(6):329-43.
~.73 (.61 - .85)
Breathalyzer (Uncooperative Patients), Serum Blood Alcohol Level
Gibb KA, Yee AS, Johnston CC, Martin SD, Nowak RM. Accuracy and usefulness of a breath alcohol analyzer. Ann Emerg Med. 1984;13(7):516-20.
.73
Fitbit Ultra Steps, Handcounted Steps
Stackpool, Caitlin M. Accuracy of various activity trackers in estimating steps taken and energy expenditure. 2013
~.73 (.44, .49, .99, .99)
Nike Fuelband Steps, Handcounted Steps
Stackpool, Caitlin M. Accuracy of various activity trackers in estimating steps taken and energy expenditure. 2013
~.67 (.17, .55, .97, .98)
Pertinacity's Fist-Sized Portions, Calories from DB
Precision
.65
Jawbone UP Calories, Portable Metabolic Gas Analyzer
Stackpool, Caitlin M. Accuracy of various activity trackers in estimating steps taken and energy expenditure. 2013
~.63 (.40, .57, .69, .87)
Calories Reported, Calories Fed
Stackpool, Caitlin M. Accuracy of various activity trackers in estimating steps taken and energy expenditure. 2013
.61
Changes in Pertinacity's Fist-Sized Portions, Changes in Calories from DB
Dynamics
.59
SAT, First-Year College GPA
Validi ty of the SAT® for Predicting First-Year College Grade Point Average
.53
Fitbit Ultra Calories, Portable Metabolic Gas Analyzer
Stackpool, Caitlin M. Accuracy of various activity trackers in estimating steps taken and energy expenditure. 2013
~.49 (.24, .41, .63, .67)
Nike Fuelband Calories, Portable Metabolic Gas Analyzer
Stackpool, Caitlin M. Accuracy of various activity trackers in estimating steps taken and energy expenditure. 2013
~.44 (.08, .47, .49, .72)
GRE Subject, First-Year Graduate GPA
A Comprehensive Meta-Analysis of the Predictive Validity of ...
.34
GRE Verbal, First-Year Graduate GPA
A Comprehensive Meta-Analysis of the Predictive Validity of ...
.24
Great Stock Picker, Future Stock Returns
Active Portfolio Management: A Quantitative Approach for Producing Superior Returns and Controlling Risk
.10

(The numbers prefixed by ~ are mean values of the numbers that follow in parentheses. The means were calculated by me, whereas the numbers in parentheses are given in the reference publication.)

Summary

The Pertinacity iPhone App asks you to eat less to lose weight. It's like other calorie counters, with two major differences:

  • (i) It's much easier to count your food with Pertinacity. You use a calorie proxy to count all food -- restaurant, packaged, and home-cooked -- the same way.

  • (ii) Your daily calorie limit is customized for you based on your recent eating habits which means you'll reduce your calorie intake little by little, at a comfortable pace.

The Calorie Proxy

The calorie proxy is this: You imagine squashing the food item you want to count. You make a fist, then estimate how many fists would be the same size as the squashed food item. We refer to this as a "fist-sized portion". You record the number of fist-sized portions your eat each day with the Pertinacity App.

Pertinacity's calorie proxy is as accurate as looking up food in a calorie table. You won't, on average, undercount (or overcount) calories with Pertinacity -- well, no more so than with a calorie table. People typically undercount calories whether they realize it or not no matter how they're reporting those calories. It's a fact of life that we accept and work around. Pertinacity's adaptive calorie limit compensates for this natural undercounting.

Like any weight loss plan or tool, Pertinacity only works if you use it, so care was taken to make it as easy for you to use as possible. While the calorie proxy is less precise than looking up food in a calorie table, the proxy also makes Pertinacity much easier to use. When people track their calories by looking up their food in a calorie table their answer is typically within 18% of what's expected. For Pertinacity's proxy that number is about 28%. Accepting this extra imprecision allows Pertinacity's interface to be dramatically simplified. This means you'll be more likely to use it, which means you'll be more likely to lose weight.

The Calorie Proxy Limit

Pertinacity always encourages you to eat slightly less than you are comfortable with. After your body and lifestyle adapt Pertinacity lowers the limit, again, to a level slightly lower than you are comfortable with. As time goes on you eat fewer and fewer calories and your weight decreases. This happens at a comfortable, customized pace.

Each day your goal is to keep your intake at or below the limit specified by Pertinacity. This limit is slightly less than the average of the past two weeks of daily calorie proxy values. For example, if you've been eating an average of 15.6 fist-sized portions over the past two weeks, Pertinacity will ask you to eat at most 15 fist-sized portions today.

That might seem like a small change, but that's ok, because smaller changes are easier to make.

After you've succeeded in eating at most 15 fist-sized portions per day consistently for two weeks, Pertinacity will change the limit to 14. It might take you longer than two weeks to stay consistently under the limit. If so, Pertinacity will wait until you're ready before lowering the limit.

The adaptive limiting method also solves the problem caused by underreporting. Consider this: Let's say I told a dieter that eating 1800 Calories/day would result in weight loss. The dieter then records 1800 Calories/day. We know from research that people tend to undereport calories by 20%, so a report of 1800 Calories/day could mean that the dieter actually ate 2250 Calories/day, which might be too high to result in weight loss.

One way to view this problem is to say that we're not comparing "apples to apples". The 1800 Calories/day limit comes from an unbiased estimate (a statisical model fit to a crosssection of the population) of the Calories/day a dieter needs. The dieter's report of 1800 Calories/day comes from the dieter and is biased toward underreporting. These two values of 1800 are like apples and oranges. They can't be meaningfully compared.

Pertinacity's adaptive limit works only with your reported numbers. When you look at the App on your phone you're comparing reported values to an average of reported values. You're making an "apples to apples" comparison.

Motivating Research

To lose weight you need to create an energy imbalance. This means you need to use more energy (Calories) than you consume to lose weight. To create an energy imbalance you could eat less or move more -- or both.

Generally speaking, solving one problem is easier than solving two, so we need to choose which problem to solve: eating less or exercising more. Pertinacity focuses on eating less because studies suggest that it's more powerful for weight loss than exercising more. Note that this isn't a statement that exercise isn't also effective for weight loss (just less so than eating less) or good for other aspects of your health.

Studies suggest that adherence to a weight loss plan improves results. In other words, for a weight loss plan to work you have to follow it. This might be unsurprising, but it is important to state because it prompts us to ask, "How do we improve adherence?"

We improve adherence by making a weight loss tool simpler to use. That is why Pertinacity uses a proxy rather than a calorie table. It is also why Pertinacity focuses on a only single weight loss method (eating less).

So, Does it Work?

Our data show that consuming fewer fist-sized portions today than a recent average is correlated to consuming fewer calories than a recent average. This implies that we can control calorie consumption by proxy.

No controlled experiments have yet been run directly on users of Pertinacity. However, Pertinacity was engineered to help you use what is possibly the most well-studied and understood weight loss "method": reducing calorie intake. Complete details of Pertinacity's research and development are available in the blog.

There's no new science in Pertinacity. There's old, boring science: eat less to weight less. The novelty of the App is, rather, in its approach to measurement and control, i.e., the calorie proxy and the calorie proxy limit.

I've had great success with Pertinacity. I've lost 13 pounds using it, and -- perhaps more importantly -- have kept it off. I use the App every day to track what I eat. In addition, I lost 5 pounds right before starting Pertinacity (using a regular calorie counter) for a total of 18 pounds since I decided to lose weight. My BMI has gone from from 25.1, "Overweight", to 22.6, "Normal".

I tell you this because I'm happy to be lighter and to let you know why I'm here writing this -- not because I place much stock in testimonials.

I do, however, think the methodology presented in the blog and summarized above should motivate you to give Pertinacity a try.

Try it. It's free. It's easy. Let me know how it goes.

Dynamics

We saw that measuring calorie intake with a conventional calorie counter (database lookup) gives an estimate of actual calories consumed with a spread of about 18% while using Pertinacity's proxy gives a spread of about 28%. Also, both measurements are biased by about 20% due to underreporting.

Pertinacity's calorie proxy is less precise than a database lookup but not significantly so. The variability of daily calorie measurements induced by using the Pertinacity proxy is comparable to the natural variability of calorie intake. In exchange for this loss of precision we dramatically improve ease-of-use. Pertinacity is much simpler to use than a conventional calorie counting tool.

This leads naturally to the question: Can we use our proxy measurements to control our calorie intake? If we measure Pertinacity's fist-sized portions every day and adhere to the suggested limit, will our consumption of calories decrease? The following analysis suggests the answer is "Yes".

Controlling Calories via Proxy

Data were collected from 28 Turkers on Amazon's Mechanical Turk. Two Turkers gave incorrect data, so there were N=26 usable data sets. Each consisted of seven days of daily records of food item descriptions along with calorie estimates and fist-size portion estimates.

These summary variables were calculated from the data for each of the 26 data sets:

F = mean of fists-sized portions consumed over the first six days
C = mean of database calories consumed over the first six days
f = number of fists-sized portions consumed on the seventh day
c = number of database calories consumed on the seventh day

We regress

(c-C) = beta1*(f-F) + eps

and find that beta1 = 129 Calories/fist (t-stat = 5) and correlation(f-F, c-C) = .58.

In terms of percentage changes:

(c/C-1) = beta2*(f/F-1) + eps

we find beta2 = .67 (t-stat = 5), and correlation(f/F-1, c/C-1) = .59.

In each case models regressed with constant terms showed statistically insignificant constant terms, so constants were omitted from the final models.

The first regression says that a change in consumption of one fist from the trailing 6-day average is equivalent to a change in consumption of 126 Calories from the trailing 6-day average. The second says that a 10% change in fists, f, from the trailing 6-day average, F, results in a 6.7% change in calories, c, from the trailing 6-day average, C.

In other words, changes in "fists" consumed effect changes in calories consumed. For the users in the data set above, a reduction of one fist/day in consumption would be equivalent to a reduction of 129 Calories/day.

The calorie proxy isn't as precise as a calorie database lookup (as discussed previously), and we see that imprecision in the second regression. If there were no loss of precision in the use of the proxy the second regression would give beta2 = 1.0, meaning any percentage change in the calorie proxy would give the same percentage change in database calories. The fact that beta2 = .67 means that if we set a fixed calorie reduction target we would only achieve .67 of that target if we measured with fists. In the data set above a one-fist reduction with beta2 = .67 is equivalent to 129 Calories/day. If beta2 were equal to 1.0 -- perfect precision -- a one-fist reduction would be equivalent to 195 Calories/day.

Designed for Ease-of-Use

Pertinacity's approach is to tune the system not to a specific calorie reduction target but, rather, to an "ease-of-use target". Pertinacity is concerned with making ongoing calorie reduction achievable rather than hitting a certain target weight on a certain date. Studies show that making a calorie counting tool easier to use improves adherence and that adherence improves results. Research doesn't seem to indicate that setting a more or less aggressive weight goal for a given date has any impact on weight loss success so Pertinacity doesn't do that.

Without a weight+date goal, what matters primarily is that calorie intake over time decreases, and the results above indicate that reducing fist-sized-portion intake reduces calorie intake.

Conclusion

Reducing calorie intake as estimated by Pertinacity's proxy -- fist-sized portions -- results in a reduction of calorie intake as measured by a calorie database.

The advantage of Pertinacity's proxy is that it is very easy to use. Ease-of-use improves adherence, and adherence improves results.

Accuracy

At the core of Pertinacity is a proxy for calories. To evaluate this proxy we need to consider precision and accuracy. The previous blog post addressed precision. In this post we address accuracy.

Recall the calorie proxy:

  • Imagine squashing the food item.
  • Make a fist. Estimate how many fists would be the same size as the squashed food item.
  • Exclude drinks known to have zero calories (water, diet soda, tea, black coffee, etc.), but include other drinks like smoothies, alcohol, regular soda, milk, juice, etc.

An evaluation of accuracy is best done in terms of bias: bias: The average value of errors in a measurement. Lower bias measure higher accuracy.

Biases in Calorie Counting

Calorie counting systems possess several biases. They still help people lose weight, but they do so in spite of the biases.

Calorie counting systems usually have two parts: (i) a measurement of the calories you consume in a day, and (ii) a limit on the number of calories you consume in a day. The limit is derived from a model like this:

Calories In - Calories Out < 0

where Calories In is the number of Calories you measures and Calories Out is given by

Calories Out = Estimated Energy Requirement (EER) = Calories expended to maintain weight as a function of age, gender, weight, height, and level of activity

Bias: EER

EER is defined in report Institute of Medicine Dietary Reference Intakes (DRI) as "the average dietary energy intake that is predicted to maintain energy balance in a healthy adult of a defined age, gender, weight, height, and level of physical activity, consistent with good health". The DRI presents a statistical model of EER based on age, gender, etc. Like any model there is some uncertainty (or spread as defined previously). The spread in the estimate of EER is about 200 Calories for males and 180 Calories for females Table 5-14.

When applying the EER estimate to calorie counting we'll get a number for Calories Out and try to eat less than that number for many days in a row. Imprecision in the estimate of that number thus appears as a persistent bias in our calorie counting system. The DRI describes the situation this way: "By definition, the estimate would be expected to underestimate the true energy expenditure 50 percent of the time and to overestimate it 50 percent of the time, leading to corresponding changes in body weight."

EER cross-sectional imprecision leads to a persistent bias in a calorie-counting system -- in Calories Out -- of around +/- 200 Calories.

Bias: Calories In

If you've ever dieted you'll be unsurprised to know that people underreport the amount of calories they've eaten. They even do it when they're not dieting. They even do it if they're dieticians (see below).

A paper, What are people really eating? The relation between energy intake derived from estimated diet records and intake determined to maintain body weight, based on the Beltsville study data, finds underreporting of calories by 18% for both men and women. This corresponds to about 550 Calories/day for men and 430 Calories/day for women.

Other studies also find that subjects underreport calories consumed. Validity of self-reported energy intake in lean and obese young women, using two nutrient databases, compared with total energy expenditure assessed by doubly labeled water. concludes "Both physically active lean and sedentary obese women under-reported TEI...". This paper measured underreporting of 23%, 30%, 39%, and 38% in the four different groups (lean and obese women using two different calorie databases) that were studied.

Dieticians did a better job than non-dieticians in Energy intake and energy expenditure: a controlled study comparing dietitians and non-dietitians., but dieticians still underreported by a little over 200 Calories/day. Non-dieticians in this study underreported by over 400 Calories/day.

Underreporting give a persistent bias to Calories In of around 400 Calories.

Bias: Activity

Activity level is specified categorically as Sedentary, Low Active, Active, or Very Active. These categories appear in the estimation formulas for EER given on p. 185 of the DRI.

Each category corresponds to a range of calories expended on physical activity. We can use the formulas to estimate the span of calories covered for a category. For example, for a man weighing 180 lb. (81.6 kg) and standing 5' 10" (1.78m) tall, we have

Sedentary   0-248 Calories, span = 248 Calories
Low Active  248-565 Calories, span = 317 Calories
Active      565-1084 Calories, span = 519 Calories
Very Active >1084 Calories

If we change the gender, height, or weight in the calculations we'll get different answers. Also, it is unclear what the span of "Very Active" is, since there is no specified or obvious value for the upper limit on the amount of calories a person can expend through physical activity.

The point, however, is that there is some uncertainty in the specification of activity level. If the example individual correctly chose "Low Active" as his activity level he would have an uncertainty in the calories expended of 317 Calories/2 = 158.5 Calories. This discrepancy is already included in the EER bias discussed above.

If he chose the wrong category he'd introduce a new, persistent bias that is not part of the EER bias. Since overreporting of physical activity has been observed, there might also exist bias in a person's selection physical activity category.

Dealing with Bias

These biases affect the quality of your calorie counting by creating a diet that, on average, has more calories than expected. Additionally the biases are persistent which means you should expect to be biased by the same amount every day. In other words, you can expect to overeat relative to your goal every day if you use the formulas directly.

Fortunately, the DRI also provides methods for dealing with bias.

Method I (Dynamic):

"This indicates that monitoring of body weight would be required when implementing intakes based on the equations that predict individual energy requirements. For example, if subjects were enrolled in a study in which it was important to maintain body weight, each individual would be fed the amount of energy estimated to be needed based on the EER equation. Body weight would be closely monitored over time, and the amount of energy provided to each individual would be adjusted up or down from the EER (or TEE) as required to maintain body weight."

Method II (Static):

"If the goal of planning is to prevent weight gain in an individual with specified characteristics, the appropriate EER equation could be used to derive the aver- age energy expenditure for the individual, and then subtract an amount equal to two times the SD. This would lead to an intake that would be expected to fall below the actual energy requirements of all but 2.5 per- cent of the individuals with similar characteristics. Using the above example for the 33-year-old, low-active woman, the energy requirement would be 2,028 – (2 × 160) kcal, or 1,708 kcal. This intake would prevent weight gain in almost all individuals with similar characteristics. Of course, this level of intake would lead to weight loss in most of these individuals."

Method I says to start with the amount recommended by the EER formula, monitor a person's weight, and adjust the calorie intake accordingly if weight is not maintained. Method II says to set the calorie intake so low that you overcome the "spread" (or imprecision) of the estimate and increase the probability of avoid weight gain or inducing weight loss.

Method I addresses weight maintenance rather than weight loss, but one could modify the suggestion to say, "If weight loss is not achieved, then adjust the energy provided down." In other words, if you're not losing weight, trying eating less. In fact, if the purpose is solely to lose weight, that adaptive strategy should work from any starting calorie intake -- not just from the EER estimate.

Pertinacity adopts Method I with the modification for weight loss just described. Additionally, Pertinacity uses as a starting point (instead of the EER formula) a measurement of your current caloric intake. In short, it says, "Try eating less than you have been and see if you lose weight." And it keeps saying that every day.

Specifically, Pertinacity recommends a maximum of one fist-size portion (FSP) reduction in your calorie intake relative to your current level of intake. Based on the data collected from Amazon's Mechanical Turk (see previous blog post for a description of the data), this reduction should be equivalent to 11% on average, and about 6%-13% for most people. Since the recommended limit on FSP is always based on the trailing two weeks of measurements this number will change slowly. (It won't decrease by 11% every day, for example. It'll start at an 11%-ish reduction then when you are successful at eating at the reduced level for a couple of weeks it'll make another reduction. The process is gradual.)

Notice that using this moving average method for recommending a limit on intake obviates the need to explicitly model calories in terms of FSP. That process would require more data from the user, so it represents another simplification and, thus, another improvement in ease-of-use. Additionally, due to the use of this method, Pertinacity does not need to collect data on activity or the other user characteristics needed to estimate required energy intake.

To deal with biases in calorie counting systems and avoid having to model calories in terms of FSP, Pertinacity employs a dynamic algorithm that works like this: Reduce your FSP intake and that will reduce your calorie intake which will, in turn, reduce your weight.

Conclusion

Calorie counting systems have several biases embedded in them, including (i) statistical error in the EER estimate of calories required (+/- 200 Calories), (ii) underreporting calories consumed (400-500 Calories), and (iii) overreporting of activity. Nevertheless, the systems work for their users, in part, because of the use of a bias-defeating method like (I) dynamically decreasing the calorie intake recommendation until the desired result is achieved, or (II) recommending a large, static, change in calorie intake.

Pertinacity uses a gradual, dynamically-changing limit on the number of fist-sized portions the user may consume. This makes the system accurate and simplifies it dramatically.