[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RE: [xmca] Wearing out truth



Mike,
I, too, found the article intriguing--enough so as to write a letter to
the editor about it. 
No doubt, it's just a matter of time before mine is selected for
publication out of the hundreds they will undoubtedly receive on this
article, but in case not, here's the comment for anyone interested. 
David

Letter to the Editor: New Yorker

Jonah Lehrer ("The Truth Wears Off," December 13, 2010) provides a
provocative array of evidence for the "decline effect," the tendency for
statistically sound scientific studies published in journals of
psychology, ecology, and medicine to fail replicability tests in a
gradual decline of effect sizes: "'It was as if nature gave me this
great result and then tried to take it back'" (p. 53). Though Lehrer
presents many snippets of possible explanation, he provides no synthesis
or evaluation of arguments. The most plausible factor is publication
bias. Because publishable results almost always have to show
statistically significant results requiring a 95% confidence level, this
means up to 5% of statistically significant findings could be
attributable to chance error. But in that case, we'd expect subsequent
replications to generally show no effect size, not a gradually declining
sequence of effect sizes. The only possible alternative explanations
seem to be metaphysical in nature: "'it was like the cosmos was
habituating to my ideas'" (p. 53). 

Pursuing the metaphysics, a more palatable explanation than a physical
universe actively interested in what we think would be the solipsistic
view that the universe we experience is a projection of our own psyche.
Thus the pattern of decline recorded in Lehrer's report actually would
index our own declining psychic investment in replicated results. If
this were the case, we'd expect it to hold, also, for hard science
results, not just results involving human response, though in a less
obvious and dramatic form. This could be tested empirically by doing
repeated micro-measurements on new physical constants as may come to be
discovered in physics or chemistry. A slight lessening in degree of
accuracy of measurement of physical constants wouldn't have aroused the
attention of scientists in the past. However, a pattern of such
declining resolution could serve to validate the solipsistic thesis that
the universe out there is what we make it-now wouldn't THAT be
troubling!  


-----Original Message-----
From: xmca-bounces@weber.ucsd.edu [mailto:xmca-bounces@weber.ucsd.edu]
On Behalf Of mike cole
Sent: Tuesday, December 28, 2010 10:33 PM
To: eXtended Mind, Culture,Activity
Subject: [xmca] Wearing out truth

If you have not seen these materials, gathered by a colleague, i suggest
you take a look. Very interesting.
mike

---------- Forwarded message ----------
From: Frank Kessel <kesfam@pdq.net>
Date: Tue, Dec 28, 2010 at 5:06 PM
Subject: Truth?
To: Frank Kessel <kesfam@pdq.net>


 FOLKS: As you may (or may not!) remember, in a message forwarding the
news about how Republicans plan to (try to) cut/reduce the 'soft'
behavioral and social sciences from the NSF budget (see below-below), I
provided the abstract for a recent *New Yorker* article on the
less-than-firm quality of at least some of our hard(er) science
findings.  Well, in the 2nd part of his annual "Sidney Awards" columns,
David Brooks talks about the same article (see below).  Better still, he
[rovides a link that enables all of us non-subscribers to access the
whole article. So here 'tis (attached). . .
Methinks that, along with the earlier-in-the-year article and discussion
re the limitations of much of WEIRD (psychological) science, this
presents us with, well, let's simply say a non-trivial challenge ot
three!  FRANK

P.S.  In case you don't know, Brooks' "Sidney Awards go to some of the
best magazine essays of the year. The one-man jury is biased against
political essays, since politics already gets so much coverage. But the
jury is biased in favor of pieces that illuminate the ideas and
conditions undergirding political events" . . . and are named for Sidney
Hook.

------------------------------------------------------------------------
------------------------------------------------------------------------
------------------------------------------------------------------------
---------------------

In earlier ages, people consulted oracles. We consult studies. We rely
on scientific findings to guide health care decisions, policy making and
much else. But in an
essay<http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer
>called
"The Truth Wears Off" in The New Yorker, Jonah Lehrer reports on
something strange.

He describes a class of antipsychotic drugs, whose effectiveness was
demonstrated by several large clinical trials. But in a subsequent batch
of studies, the therapeutic power of the drugs appeared to wane
precipitously.

This is not an isolated case. "But now all sorts of well-established,
multiply confirmed findings have started to look increasingly
uncertain,"
Lehrer writes. "It's as if our facts were losing their truth: claims
that have been enshrined in textbooks are suddenly unprovable."

The world is fluid. Bias and randomness can creep in from all
directions.
For example, between 1966 and 1995 there were 47 acupuncture studies
conducted in Japan, Taiwan and China, and they all found it to be an
effective therapy. There were 94 studies in the U.S., Sweden and
Britain, and only 56 percent showed benefits. The lesson is not to throw
out studies, but to never underestimate the complexity of the world
around.

------------------------------------------------------------------------
------------------------------------------------------------------------
------------------------------------------------------------------------
---------------


*From:* Frank Kessel [mailto:kesfam@pdq.net]
*Sent:* Thursday, December 23, 2010 3:14 PM
*To:* 'Frank Kessel'
*Subject:* News for the New Year: Cutting out the social science funding
from NSF

 So the battles begin  . . . again = the wacky Wisconsin Senator's
"Golden Fleece Award" redux . . . but, alas, probably with far greater
political force.

As for the, um, certainty of the, uh, hard sciences -- See attached.
(Will zip over complete article as soon as one or another subscriber
gains
access!)


------------------------------

>Representatives Eric Cantor's and John Boehner's 2009 
>proposal<http://republicanleader.house.gov/UploadedFiles/06-04-09_Savin
>gs_Proposals_For_President.pdf>to President Obama seeking to cut in 
>half the NSF's $198 million allotted
for awards in behavioral and social science. "Unlike NSF's other hard
science programs (such as engineering and biological sciences)," Boehner
and Cantor wrote in 2009, "these soft science programs are often more
controversial and less directly related to NSF's core mission."

*Picking on Social Science *

December 21, 2010  Inside Higher Ed

A bid to question the merits of federal funding for social and
behavioral science research may be failing to capture the public's
attention, even as it signals that larger and more polarizing battles
over science, federal policy and money could lie ahead.


On Aug. 13, Rep. Adrian Smith, (R-Nebraska) posted a clip on
YouTube<http://www.youtube.com/watch?v=LSYTS-nRt4o>announcing the launch
of YouCut <http://republicanwhip.house.gov/YouCut/Review.htm> -- an
attempt to get citizens to crowdsource ways to cut the federal budget.
In YouCut's first foray, Rep. Smith invites viewers to share their
impressions of grants for research that have been awarded by the
National Science Foundation.
After praising the NSF for supporting discoveries in the "hard" sciences
(typically math, engineering, and the physical, natural and
computational sciences), which have spurred economic growth, he cites
what he sees as two of the NSF's more suspect awards.

"University academics received a $750,000 grant to develop computer
models to analyze the on-field contribution of soccer
players<http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0838564>
,"
he says (it's actually a Northwestern University project led by
engineering and business professors to develop strategies to better
assemble effective teams in virtual communities). Rep. Smith also
mentions a $1.2 million award to model the sound of objects
breaking<http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0905506
&loc=interstitialskip>for
use by the video game and movie industries (the project involves Cornell
University computer scientists finding ways to change how sound is
manufactured in interactive virtual environments).

Attacks of this nature -- which tend to target perceived intellectual
and cultural elites -- often gain traction during periods in which
Republicans control at least one house of Congress, as they are about to
do. This position of power enables them to hold hearings, call for votes
on specific projects and exert some control over the purse strings of
federal agencies.
Such attacks already have begun. Republicans led a successful effort
earlier this month to pressure the Smithsonian Institution to remove a
work of art from the National Portrait Gallery's exhibition, "Hide/Seek:
Difference and Desire in American
Portraiture,"<http://www.insidehighered.com/news/2010/12/06/katz>for
its perceived anti-Christian imagery.

"We've been down this road before," said Howard Silver, executive
director of the Consortium of Social Science Associations, who recited a
litany of efforts by politicians of both parties to question science
research. Often cited as the originator of this type of Congressional
activity is Sen.
William Proxmire (D-Wisconsin), whose Golden Fleece awards drew
attention to wasteful spending -- with scientific research one of his
favorite targets.
More recently, Pat Toomey, then Republican representative and now
senator from Pennsylvania, tried in 2003 to block National Institutes of
Health-backed research on sexual
risk-taking<http://www.iub.edu/%7Ekinsey/about/risk-research.html>.


Many times, these salvos -- in which politicians pounce on
silly-sounding research projects, often without understanding their
underlying purpose -- have ended up backfiring. Proxmire once ridiculed
federal money spent to study the screwworm fly, but later
conceded<http://www.ucop.edu/pres/comments/gfleece.html>that this
research led to extremely effective efforts to eradicate the nasty pest
that devoured the flesh of
cattle<http://books.google.com/books?id=UGjtXW4eoNkC&pg=PA75&lpg=PA75&dq
=Proxmire+Screw+worm&source=bl&ots=l6rspcnZqT&sig=ZwcYSadsMzP-V5s0ACFp3h
DNlT0&hl=en&ei=sqcLTaXcB8OB8gbM-IWZDg&sa=X&oi=book_result&ct=result&resn
um=7&ved=0CDkQ6AEwBg#v=onepage&q&f=false>.
Mark Sanford, as a U.S. representative from South Carolina, sought to
freeze NSF funding. He staked much of his argument that the agency
mismanaged money on the fact that it supported research into
ATMs<http://www.newscientist.com/article/mg15921536.200-not-that-kind-of
-poker.html>,
a term he mistakenly thought referred to automatic teller machines.
Instead, it describes asynchronous transfer mode, a telecommunications
innovation that enables data, voice and video to be transmitted in one
data stream.

YouCut's scrutiny of the NSF -- which Silver described as "interesting
political theater" -- is more explicit than past efforts in dividing the
physical and natural sciences on the one hand from the behavioral and
social sciences on the other. Silver said he was troubled by
Representatives Eric Cantor's and John Boehner's 2009
proposal<http://republicanleader.house.gov/UploadedFiles/06-04-09_Saving
s_Proposals_For_President.pdf>to
President Obama seeking to cut in half the NSF's $198 million allotted
for awards in behavioral and social science. "Unlike NSF's other hard
science programs (such as engineering and biological sciences)," Boehner
and Cantor wrote in 2009, "these soft science programs are often more
controversial and less directly related to NSF's core mission." That
statement was written before both men were slated to assume much more
power
-- Boehner as Speaker of the House and Cantor as Majority Leader. And
their interest in finding areas in the budget to cut has remained
undimmed.

Rep. Smith's address on YouTube continues to distinguish between the
value of these branches of science. While the two examples he offers as
evidence of questionable research blend different scientific
disciplines, Smith frames NSF's most worthy work as being in engineering
or the physical, natural and computational sciences. He asks viewers to
"help us identify grants which do not support the hard sciences or which
you don't think are a good use of taxpayer dollars."

Experts and advocates for science in general, and social science in
particular, have questioned this separation. "It's sort of an easy way
to make political hay," said Silver, noting that social and behavioral
sciences account for a small fraction of the NSF's annual awards (about
3 percent of the agency's $6 billion total budget, according to the
agency). "Is there a business in this country that doesn't need to
understand human or societal behavior? This whole business of saying
these aren't as useful or as important as natural or physical science
doesn't make sense to me."

Similarly, Barry Toiv, vice president for public affairs at the
Association of American Universities, said his organization and its
allies will need to make a forceful case that social sciences are as
important an investment as the the natural and physical sciences, while
acknowledging that no formal campaign was in the works. Toiv cited the
University of Michigan's surveys of consumers
<http://press.sca.isr.umich.edu/press/about_survey> as an example of how
social science research benefits public policy. Another
example: the research now being conducted at Massachusetts Institute of
Technology
<http://web.mit.edu/newsoffice/2010/profile-pentland-1101.html>on
understanding nonverbal cues. The results could help troops in
Afghanistan better read the body language of people with whom they
cannot communicate verbally, he said. "A particular project may sound
funny or irrelevant, but you never know where that's going to lead,"
said Toiv.

To a great extent, this argument has been won within the NSF, based on
its commitment to fund cross-disciplinary
research<http://www.insidehighered.com/news/2010/08/16/nsf>,
which was articulated in August by the agency's assistant director for
the social, behavioral and economic sciences.

Among policy-makers, it's a different story, albeit one with some new
technological, rhetorical, disciplinary and political wrinkles. YouCut's
use of crowdsourcing marks a "clever" change in approach, said Al Teich,
director of science and policy programs for the American Association for
the Advancement of Science. "This is a high-tech twist on an old story,"
he said. Teich and others hope that it could be fruitfully deployed by
scientists and their advocates to make the case for their research. "It
might be a useful technique to engage the public in this sort of
conversation," he said.

April L. Burke, founder of the lobbying and consulting firm Lewis-Burke
Associates, also thought the effort to focus attention, through YouCut,
on the NSF could serve as a valuable opportunity for the sorts of people
and institutions she represents -- research universities and scientific
organizations. Scientists and their advocates would do well to see the
latest scrutiny not as a blanket condemnation of science and cause for
offense, but as an invitation to make the case as to why the NSF should
fund social science, she said. "I'd rather see us launch a positive
campaign and talk to members of Congress and get them comfortable with
social sciences and physical sciences," said Burke, "and save our angry
powder for when we're really under attack."

It is not clear that the bid to draw close scrutiny to the NSF has had
the desired effect. While the clip was posted in August, the effort only
started to generate more widespread attention in the past two weeks,
from such outlets as
*Wired*<http://www.wired.com/wiredscience/2010/12/nsf-youcut-review/>and
*USA
Today*<http://www.usatoday.com/tech/science/columnist/vergano/2010-12-05
-politics-science_N.htm>.
YouTube tracking data indicate that Rep. Smith's address had generated,
as of press time, about 12,200 views over the past four months. Twelve
viewers indicated that they like the page, while 195 dislike it.
Responses on Twitter to Rep. Smith's address have not been particularly
kind, either.
Some have called it a "witch hunt," "catastrophically dumb," and
dangerous in a country with a high rate of scientific illiteracy.

Rep. Smith's office referred to Rep. Cantor's office questions from
*Inside Higher Ed* on the number of awards that have been challenged.
Rep. Cantor's office did not respond to several calls and e-mails --
including a request to clarify whether *any* citizens had raised
objections. Maria Zacharias, a spokeswoman for the NSF, said she was not
aware of any other grants being called into question by members of the
public, though such a lack of result is not necessarily unexpected.

Several observers thought the real significance of the YouCut episode
was that it offered a preview of upcoming efforts to trim federal
spending on NSF and other agencies back to 2008 funding levels, and of
looming investigations over climate science or energy policy. Teich's
group, the AAAS, has offered a sober but slightly hopeful
view<http://www.aaas.org/news/releases/2010/1201post_election_energy.sht
ml?sa_campaign=Internal_Ads/AAAS/AAAS_News/2010-12-01/jump_page>of
the shifting landscape since last month's election, and suggested that
consensus in science funding, climate change and energy policy could
still be built.

Toiv, of the AAU, declined to see the prospect of future conflicts over
research in strictly partisan terms. He said political pressure over the
deficit was bound to place discretionary domestic spending, which is the
source of NSF's money, under the microscope. "There was going to be a
problem no matter which party won," he said. But, he added, it remains
important for advocates of scientific research to highlight the economic
argument to support future funding. "We cannot hope to have the kind of
economic growth we need to address the deficit issue without, over the
long-term, making these investments now in basic research," said Toiv.
"These are investments. That's what needs to be clear."

- Dan Berrett <dan.berrett@insidehighered.com>
__________________________________________
_____
xmca mailing list
xmca@weber.ucsd.edu
http://dss.ucsd.edu/mailman/listinfo/xmca