Juking the Stats

Over-reliance on data makes the public go blind

I struggle with the widespread adoption of data these days. This may sound strange since I’ve always loved math and I’ve worked with data my entire career (and still do). The White House now has a Chief Data Scientist, and demand for people in my field seems to have exploded over the past few years. But I’m concerned about how quickly data as a practice is spreading across companies, not because it’s a bad trend in and of itself but because of how some people are using metrics. Let me explain.

On the one hand, it’s encouraging to see more organizations embrace measurement and accountability. It makes sense to use logic and facts whenever possible. It makes sense to monitor how a marketing campaign performs or how the sales team is doing against quota or how engaged consumers are with a product. It makes sense to keep investors, shareholders and team members on the same page with respect to the business.

On the other hand, data integrity is an increasing battle. Whether intentionally or not, misleading studies, analyses and stats are thrown around all the time now (esp. online). Some organizations fudge numbers to manipulate public perception or push dishonest agendas. As data becomes increasingly part of our language and our DNA, I can’t help but wonder if we have the proper checks and balances to monitor what’s really going on inside those spreadsheets. Are the assumptions sound? Are they concluding causation or correlation? How confident are they of what they’re reporting?

Furthermore, I have seen many teams obsess over hitting certain quantitative targets instead of focusing on the bigger picture. With an increased focus on team goals, some people take shortcuts or do things just to boost their numbers up. One of the biggest challenges with business today is to align incentives properly across teams and individuals for collective impact rather than personal gain.

Whenever I try to talk about the risk of holding people accountable for showing good numbers, I like to reference The Wire. It’s my all-time favorite TV show and it does a great job of displaying how we have a systematic issue across all sectors. Whether it’s politics, the police, education, the media, or corporate America, everyone is under pressure to show improvement, upward trends, and “strong leadership”.

A scene from The Wire (script via IMDb):

Roland ‘Prezbo’ Pryzbylewski: I don’t get it. All this so we score higher on the state tests? If we’re teaching the kids the test questions, what is it assessing in them?
Grace Sampson: Nothing. It assesses us. The test scores go up, they can say the schools are improving. The scores stay down, they can’t.
Roland ‘Prezbo’ Pryzbylewski: Juking the stats.
Grace Sampson: Excuse me?
Roland ‘Prezbo’ Pryzbylewski: Making robberies into larcenies. Making rapes disappear. You juke the stats, and majors become colonels. I’ve been here before.
Grace Sampson: Wherever you go, there you are.

“Juking the stats”

Juking the stats happens all the time. Part of the reason may be that people want to give the public what it wants: reassurance that things are great. Here are some classic reasons we read or hear about:

• We’ve already committed to the story we want to tell, and we just need to find numbers to justify the budget
• We need to instill confidence in our business to the public/investors/shareholders
• People’s bonuses depends on it and I can’t afford to lose more people
• It’s fight or flight, and I’m not going to let me and my team get fired over this
• We know it’s true in our gut. We just need numbers to show our gut’s right
• Our hands are tied

So what could we do to help?

Whether or not you work with data, there are 5 areas that we should be advocating for in our respective fields in the name of transparency and organizational health:

• Continuous and legitimate training on how to interpret data. When I worked at Google, I was blown away by the geniuses I got to call my peers. As an Economics major, I always viewed Hal Varian’s team as THE team when it came to quant thought leadership. I wish they would have done more training on analytics & statistics across the organization, from Engineers to Analysts. As data becomes more widespread, it’s more important than ever to make sure people are interpreting data correctly across teams and executive members.
• Accountability with numbers and assumptions. How many professionals have you sat across the room from who exude confidence and polished presentation skills? How many of them have you legitimately challenged? Every model relies on some assumptions, and it’s important to have checks and balances in place to monitor those to ensure realistic projections.
• Aligned incentives for the collective good. Some companies add a company performance multiplier to individuals’ bonuses, meaning the better the company does the better the individual’s paycheck becomes. There are various compensation models out there, and the more they align with collective impact the better teams will work together against a collective purpose.
• Data -> Insights -> Story. Rather than obsessing over telling a specific story, I find that the best outcomes often happen because data highlights an insight, which then introduces an interesting story about consumers or the industry. Trying to fit numbers into a pre-crafted story is often manipulative and deceptive.
• Decision-making. There are some instances when data is not enough to justify a final decision, particularly at small companies where there isn’t a lot of data to look at yet. There’s a difference between using data and needing data to move forward. Sometimes one needs to take a leap of faith and take risks. And this is where the Art & Science in every job comes into play.