Thanks for letting me use the lab on the weekend, Dr. Sherman.
Of course, Winnie, we want all the students
competing in this year's West Haven Science Competition
to feel supported.
How's your research project coming along?
Oh, it's very promising.
I'm gunning to win this thing.
Oh, great, well, good luck.
I'd offer to stay and help but Sunday is my fun day.
So, I'm just gonna leave you here to use the lab
completely unsupervised.
Thanks.
Let's get to work, Carl.
If my hypothesis is correct,
my project will blow everyone else's out of the water.
And then I'll get into the university of my choice...
and be on the path to becoming a groundbreaking scientist
just like my heroes.
I'll follow in the footsteps of Marie Curie
and Mimi Phipps Clark.
And do research to benefit mankind.
Because Carl, in a world filled with ambiguity,
science is always right.
Actually... Science gets it wrong all the time!
Mother physics!
Wait, are you Adam from TV?
Oh, you watch the show.
Yeah, when "Myth Busters" isn't on.
Now those guys are smart.
Well, wanna do the honors?
Sure.
Hi, I'm Winnie Jones and this is Adam Conover,
and I'm on "Adam Ruins Everything."
Uh-oh.
Corrected & Synced by Bakugan
Adam, I've seen your show.
It's about proving things wrong.
But you can't do that to science.
Science is how we find out the truth about the world.
Yes, it is.
But the way we do science is often flawed.
Well, not here, it isn't.
Carl and I are working on a very exciting, very solid study
about how Vitamin D affects human memory retention.
Oh.
Yes, you are a good little test subject.
Actually, if you're trying to learn about humans,
he's a terrible little test subject.
Sorry.
What? Everyone knows testing on rodents
is practically the same as doing actual human trials.
Yes, everyone "knows" that.
Too bad they're wrong.
Cool! I've always wanted to be a B-A-L-B-C-J
laboratory mouse.
But what are we doing here?
The same thing we do every night, Winnie.
Ruining a common misconception.
Rodents make up 95% of all animals
used for biomedical research in America.
That's 20 to 30 million mice a year.
Whoa, they must be such effective test subjects.
Actually, they're not.
Out of all clinical trials based on mouse models,
the results don't apply to humans over 80% of the time.
Well, time to file these.
And those failed trials
can cause hundreds of millions of dollars a piece.
And this drives up the cost of development.
Meaning we get fewer new drugs.
Yeah, we're out of funding.
Well, then tell the cancer patients
they'll have to wait, Pete!
But how can that be?
I read mice share 97% of DNA with humans.
True, but that 3% makes a big difference.
Mouse metabolisms are seven times faster.
We have different inflammation reactions.
And, you know, people are over 3,000 times bigger.
And our bodies can have
very different reactions to medicine.
In one extreme example from 2006, a new cancer drug
was tested in mice and passed with flying colors.
But when human subjects were injected with just
1/500th of a mouse dosage,
within hours, all of the volunteers
went into catastrophic, multi-organ failure
and had to be rushed to the ICU.
Oh.
The truth is, testing on mice
just doesn't tell us as much about humans as we think it does.
Jeez Louise.
So, why do we even use mice in the first place?
Because they're cheap and convenient.
Compared to other animal test subjects,
mice are a bargain.
And they're great in a lab setting because they're easy
to tame and store and they breed like crazy.
Congratulations!
It's 100!
Rodents are also super easy to genetically customize.
Want a mouse with no immune system?
Or how about one that glows in the dark.
Extra tumors strike your fancy?
You can have whatever model of mouse you want.
Finally, the Animal Welfare Act, the only federal law
that covers testing on animals doesn't protect mice at all.
Oh, yeah,
we can never get away with this on a rabbit.
So, even though we know testing on mice is a problem,
we keep doing it anyways 'cause it's easier?
And at this point,
it's almost impossible not to.
Ah, Winnie, this is...
Dr. Azra Raza,
Director of the Myelodysplastic Syndrome Center at Columbia!
I traded six Albert Einsteins for her.
Thanks, Winnie!
And thanks for keeping me in mint condition.
Medical science is almost entirely built
around mouse testing.
Researchers have been using mice
since the 1800s.
And students are taught on them
from day one of school.
And most other testing methods
are prohibitively expensive.
You quite literally cannot do a study without mice.
At the moment in the United States,
the FDA will almost never approve a trial in humans
unless researchers have done it in rodents first.
The real problem is cancer research has become
so mouse centric, we cannot replace rodent testing
without overhauling the whole system.
Which is a shame.
Because if we want to cure cancer in humans,
we should be studying human cancer cells.
We are wasting valuable resources and time on mice.
Hey, come on! This is a family show.
Thanks, Doctor Raza.
You're welcome.
And here's another tip.
If you put me in your bike spoke,
it will sound like a motorcycle.
Now my whole project isn't useable.
Well, you don't have to throw out all of your work.
Nope! I am gonna start a new project from scratch.
I mean, might as well get used to it.
When I'm a scientist,
I'll be doing nothing but science all day.
Not quite.
Want me to tell you about it?
Coou have found a cool kid to mess with?
You know, Winnie, the reality working as a scientist
might not be quite what you expect.
Very funny, Adam.
You're a real Humphry Davy.
The chemist who discovered the effects of laughing gas? Hello?
I know exactly what life as a scientist will be like.
I'll spend my days investigating the big unanswered questions.
I'll follow the research wherever it leads,
beholden to nothing and no one
but the pursuit of pure scientific truth.
That's a beautiful dream.
But in reality, what science you get to do depends a lot
on who gives you money to do it.
'Cause if you wanna pursue scientific truth,
first you gotta find someone willing to pay for it.
Around two-thirds of basic research is privately funded.
But here's the problem.
The priorities of the people handing out the funding
often affect what research gets done.
Welcome to "Smart Tank."
Today the Smarts will hear a proposal from Winnie,
a young researcher competing
in the West Haven Science Competition.
Hello, Smarts.
Uh, today, I'm asking for funding
for my biomedical research.
Here's my proposal.
You'll find it's very exciting and worthwhile.
First up, a major funder of research,
corporate America.
Winnie, we might be able to help you.
I just have one question.
Is this gonna help us sell more candy bars?
No... Why?
Well, corporations and trade groups
tend to fund research that will make them money.
In 2011, the National Confectioners Association
paid for a study
that implied eating sweets actually makes kids skinnier.
If you can't provide that kind of return
on investment, I'm out.
Well, I see plenty of upside.
Your research can help us develop new drugs.
Really?
Sure.
And if your results make our drugs look good,
we'll publish them.
Otherwise we'll bury them
so the results will never be seen.
Yeah, they actually do that.
Well, that's wrong.
I won't be a part of it.
Oh, an idealist. I'm out.
Well, there's two Smarts left.
Okay, another major source of funding is philanthropy.
Winnie, I apologize for my robber baron friends here.
You and I, we're in it to help people.
See? This lady gets me.
Sure, and philanthropic donors fund some worthy research.
But they're also more likely to give money
to causes that have affected them personally.
You have to use this to find a cure for diamond allergies.
Or maybe I could study diabetes or sickle cell anemia?
Oh, sorry, sweetie, I'm out.
This is a travesty.
Isn't anyone willing to fund objective,
independent, unglamorous research?
Yes, there is one final funding source.
Public funding.
Also is Mark Cuban here
because I would, like, die if I met Mark Cuban.
Because government funding isn't controlled by profits
or the whims of one person, it supports the unsexy
yet vital basic research that all science builds upon.
And it's led to advances like...
Heart rate monitors, the Human Genome Project,
audio recording, water purification, satellites,
solar panels, super computers, GPS and the internet.
Just to name a few.
Even Google got its start with a government grant.
But you won't see that on the first page of search results.
Wow, sounds like public funding is awesome.
Finally a funding source with no downsides.
Well... there is one little wrinkle.
Federal science budgets are controlled by politicians.
I hereby award you, Winnie, this government grant for...
What's that, Senator?
You'd rather have tax cuts than scientific knowledge?
And there's a powerful lobby that won't like the research?
And you don't know enough about science
to care about the consequences?
Sorry, I'm out.
No! This work is worth the money.
Ask any scientist, they'll tell you.
Sorry, politicians tell me what to do, not scientists.
Maybe not the best system.
But science needs the money.
Yes, it does.
Private funding will never be enough to support research
in the public interest.
The kind of long-term careful science that doesn't generate
profits or glory, but improves people's lives
and increases our understanding of the universe.
We need public funding for science.
But in the past 50 years,
the portion of the discretionary budget
that government spends on research and development
has been cut nearly in half.
And with less funding,
there's more competition for fewer resources.
In the past two decades,
30% of National Health Institute grants got funded.
But today, it's about 17%.
No, no, no!
And from 2010 to 2013, the U.S. had the biggest
three-year reduction of scientific research
and development since the end of the space race.
But if it's so hard to get funding,
then how do we do science?
Well, sometimes we just don't.
In a recent study of 11,000 scientists,
half of them said they've had to stop doing
valuable research partway through
because of economic pressures.
Who knows what great advancements we lost out on as a result.
As much as we'd love to believe that the progress of science
is determined solely by the best and brightest,
decisions about which science gets funded
have always been controlled by non-scientists.
It's all about who holds the purse strings.
That may be true but science always triumphs in the end.
Look at all these journals.
Great, factual peer reviewed research is published every day.
Ah...
- Ugh. - What?
Nothing, just that a surprising number of scientific studies
do not... hold up.
- Oh! - Excuse me.
I'm using those studies for my project.
Go ahead, but there's a good chance
a lot of those studies are wrong.
Adam, please, when a study has been published,
that means its data is robust and accurate.
Winnie, if you'd ever read "US Weekly", you'd know
that just 'cause something is published,
that doesn't mean it's true.
What's "US Weekly?"
Wow, you are nerdier than I am.
Studies can be flawed for a number of reasons.
If the sample size was too small,
the luck of the draw could've skewed the results.
We should test this on 10,000 people but...
I only know ten.
That's okay, making friends is hard.
Or maybe there was a confounding variable.
Like, if the subject was using another product
that could affect the results
and the researchers didn't account for it.
Oh, wow, this pill is really
boosting our subject's resistance to sunburn.
And something, very rarely,
unscrupulous researchers will fake their results.
What if we just make up numbers
that show that this pill cures cancer?
Hello? Nobel Prize committee?
I've got some data you've gotta see.
Sure, so we just need a way to find out
which studies are accurate.
We already have one. And that's to reproduce them.
To show the results weren't just some fluke,
someone else has to do another study
that tests the same question
to see if they can get the same findings.
It's like double checking the answers.
Curse, my results were way different.
And here's the problem.
Since studies are usually reproduced
after they're published,
that means they have a chance
to influence other research or even become famous
before we find out they're wrong.
For instance, remember that famous study on power posing.
Uh, it's only one of the most watched TED Talks of all time.
It proved that by standing in the power pose,
your body produces hormones that give you confidence.
Well, no one's ever been able to reproduce those results.
What?
Or how about that series of studies
from the '70s and '80s
that showed you can actually make yourself feel happier
just by smiling?
Well, that makes sense, right?
Sure, except that in 2016, almost 20 labs tried
to replicate it and not one of them
was able to reproduce those findings.
Holy Hawking!
If that's true, then how many other studies might be wrong?
Oh, a lot of them.
In fact, many scientists say that we're in the midst
of a reproducibility crisis.
Like a good type of crisis, right?
One research team just replicated 100 famous psychology
studies but found they couldn't reproduce about 60% of them.
Another revisited 67 major drug studies
and found that about 75% didn't match their results.
And another team zeroed on 53 recent cancer studies
and couldn't reproduce 47 of them.
But those numbers are so high!
If science gets it wrong that often,
then we should be reproducing every study.
That'd be nice, but it probably won't happen.
There... and that's how you fold a fitted sheet.
This is Professor Nosek. of the University of Virginia.
And the co-founder of the Center for Open Science.
Adam's right, Winnie.
Replication studies are not getting done.
And it's because scientists don't have enough good reasons to do them.
Wait, what do you mean?
To succeed in their careers, scientists have to publish
and get attention for their research.
And that's more likely to happen
if their studies have new and flashy conclusions.
My study found that Fidget Spinners give you six-pack abs.
And the pressure to get these kinds of results
sometimes leads researchers to cut corners.
I didn't need test subjects.
I just asked my bro, Kevin.
He's super ripped.
On the other hand, replication studies make for good science.
But they don't get as much attention.
People just aren't interested in studies
that don't find anything new.
And I replicated a 2003 study
on classification of fine-grained polypeptide folds.
The results held up.
Hey, her work is just as important.
Yeah, but it isn't as exciting.
Science is competitive.
There aren't enough jobs or grant dollars for everyone.
So, if you were trying to get people excited about your research, what would you do?
Follow-up on someone's prior findings?
Or try to find something new and amazing?
I, uh, also did a study that found...
candy cures leukemia!
These incentives to publish lead to a lot of unreliable research.
One estimate suggest that in the U.S.,
we spend $28 billion a year on biomedical research
that can't be reproduced.
That's such a waste!
Well, flashy findings can be useful
for stimulating new areas of discovery.
But we need to change the incentives so that researchers
are rewarded for producing results
that are interesting and reproducible.
Thanks, Professor Nosek.
No problem.
Now if you'll excuse me,
I'm gonna go use the emergency shower.
I save a ton on my water bill.
Oh, thrifty.
So, even if my research gets positive results,
I still have to find someone to reproduce it,
but no one will because they only care about
their own careers?
Wait, what are you doing?
You're right, Adam.
Science sucks, I'm done.
I'm gonna go drink some Red Bull in the woods
or do whatever cool kids do.
Oh, wait!
I was about to tell you why science is actually amazing!
I thought the structure of my show!
You forgot this.
I don't need it.
I'm quitting the competition.
You know, I used to love that no matter how chaotic
and uncertain the world felt,
I could at least count on science to always be right.
But you showed me, it's not.
Winnie, do you know why science is so awesome?
It's the field of human inquiry
that embraces mistakes and learns from them.
Scientists don't just assert what they think is true.
They test their hypotheses against the empirical evidence.
And if it turns out they're wrong,
they admit it and try again.
The fact that science seeks out its own errors
and corrects them is its greatest strength.
But how do we fix what's wrong with science itself?
With more science. With more science.
When I learned about the reproducibility challenges
in science, I thought we need to study this problem.
So we started The Center for Open Science
to investigate why is it that researchers are not producing
reproducible science even though they want to?
So, we built software
so that researchers can share their data more easily
and be more transparent in their methods.
We also work with journals and societies and funders
to align the incentives so that it's in researchers interests
to be more open and reproducible in their work.
And when I first emigrated here from Pakistan,
I had no idea how much research in the U.S.
rested on flawed mouse testing.
To combat this problem, I started to collect human tissue,
and this is how I found
a novel treatment for pre-leukemia.
- Whoa. - So, if something in science is broken
like mouse models or faulty studies,
it doesn't mean we failed.
It means we need people like you to find the solution by using...
Science!
Yes!
You're right!
I have to use science to fix science.
Eureka!
Thank you, guys.
And finally... Winnie.
And your project is...
Wait, where is your project?
Dr. Sherman, I didn't do a project.
I did 19.
Because for my experiment,
I replicated everyone else's research,
and I found that all of their conclusions were...
irreproducible.
Wow, well, that doesn't quite follow the project guidelines.
But, nice work, Winnie.
Sorry, everyone else.
You suck.
Well, I didn't do it for the glory.
I did it for science.
Come on, Adam, I'll show you my new research proposal.
I got some sweet new binder tabs.
Ooh, I love office supplies.
The card moved.
Did anyone else see that?
Holy...
Hello, we're here on set with Brian Nosek
because I have way more questions for him
about the reproducibility crisis in science.
Thanks for having me.
Is the crisis real and is it something
that we should all be worried about?
Well, it's at least a crisis of confidence.
And it's, uh, something that emerged when researchers
across a variety of disciplines realized
that the published literature, what we thing we're doing
and producing and finding isn't as reliable as we think it is.
And so, can you walk us through a little bit
of how the fields realized this was happening.
Well, the oddity is is that we've known the challenges
for reproducibility for 60 years.
And it hasn't gotten a groundswell.
But that has shifted over the last ten years
because perhaps the internet helped make it easier
to evaluate lots of different papers at the same time.
Perhaps because of a couple of very dramatic findings
that there has been a convergence.
And so it has become a thing.
Now that we know about it, we can do something about it.
That's exactly right.
Science is at its best
when it is constantly self critiquing
of looking at where it's limited,
where it's not finding things that are reliable
and looking for ways to do it better.
Every scientific claim is wrong in some fundamental way.
And the process of discovery in science is figuring out
where our current models of the world are wrong
and how we can make them less wrong.
I'm so fascinated by that. Can you elaborate on
every scientific theory about the world
is wrong in some way. In some way.
If we had a perfect model of the world,
it would be as complex as the world.
How it is we describe how the world works,
how people interact, how atoms interact,
whatever it is, is a simplification.
- It is a way to try to describe... - Right.
...lots of different things that are happening in simple terms
that make predictions about next things that will happen.
Right.
But our predictions are always imperfect.
They can't quite get exactly the phenomena as we study them.
So, they're wrong. Right.
But wrong doesn't mean, oh, that's useless.
It's not information that we need.
It's not information that we can use.
Newton's Laws of Mechanics
are the most successful theory ever, right?
We use them to build bridges and skyscrapers...
I learned about them in high school, yeah.
And they're wrong, right?
Einstein showed how they are wrong
when things are moving very fast.
So, it isn't that something being wrong means,
oh, can ignore science.
It means there's always something more to learn.
So, that leads me really nicely to my next question which is...
Is this a reason that people
should stop believing or trusting science?
You know, should the audience go home
and every time they read about a new study say,
oh, that's probably never gonna be reproduced
and so I'm not even gonna think about it again?
One of the big challenges of scientific knowledge
is that is accumulates very slowly.
So, no one study is definitive.
Drinking coffee is bad for you.
No, drinking coffee is good for you.
No, drinking coffee's bad for you.
Well, make up your mind, scientists.
It's those claims that are initially very exciting.
But then survive confrontation over time
are the ones that survive and we have confidence in.
So, as a reader of the newspaper of reporting scientific results,
one thing to look for is, is this the first finding?
Is this something brand-new?
Uh, or is it the accumulation
of a lot of different pieces of evidence?
That's a very easy signal for whether that's a credible claim
or a reliable claim that we know a lot about it
versus something new and possibly exciting.
Yeah.
The reason to trust science
is because it doesn't trust itself.
Ahh. Right?
It's that science is always questioning its own findings
which makes it the most reliable way to discover things.
Thank you so much for being here.
Thanks for having me, Adam.