Thanks for posting this, I think it's an important contribution to
exploring the question of how people think in real life practice and of how
valid is testing
By the way, how are you? Are you writing now...what about?
In looking at your assumption about the validity of the test question
>The assumption of the test makers was that the test accurately sampled the
>>work and that success at the test would match and predict sucess at work.
>>
>> First question: is this true? It probably is -- I think we can assume that
>>this test has been used enough and its content is critical enough so that
>>its validity has been confirmed: people who do well on the test actually do
>>know the material.
I try to think of my own experiences with such test/work situations (for
example, drivers' manuals and written tests for state license, or truck
driver's written exam compared to real-life driving), but I realize that I
was trained in high school to beat the tests with that special
"test-taking" strategizing that Lily W Fillmore discovered in middle class
white and Chinese kids in Oakland schools...so I can't relate to your
subject's particular problem, except that I always examine multiple choice
questions from a critical stance (both in my own situation like on the GRE
and the NTE, and in my students' situation on such things as the CTBS) But
I have always managed to abstract the test questions from real-life, to
tell myself "This is not a search for the truth...just pick out the 'best'
answer, even if real life is more complicated" I think that the fact that
successful workers have managed to pass the exam is not necessarily a valid
selection mechanism, in other words, how many successful or potentially
successful workers did not pass because they didn't understand the testing
game? That's your point about
>But can there be other people, for
>>whom the test is NOT a valid sampling of the work, could could also do the
>>work well and be successful?
Here's the crux of the problem, and so I'd like to know more about how your
subject approached the test questions. Your sample question example
started in the right direction, but you cut it off by saying that he gave a
global description of the yard, and approached the right answer without
exactly hitting it. Maybe you don't know enough about rail yards to
qualify as a good judge of his thinking, but I think that's where good
research should go in order to get out of the "hall of mirrors." Despite
my test-taking savvy, which "spoils" me as a test subject (but maybe makes
me a good control subject?), I have had many different jobs in which I
realized that abstracted slices of reality don't quite capture the nature
of the whole. Many safety procedures break down under the vagaries of real
life happenings...for example, the high incidence of alcoholism in the
railroad industry might make the prescribed safety procedures insufficient,
and workers concerned with safety might need to take additional measures
that could cost money in order to ensure real safety (like shutting down a
section of track so that the worst fuck-up still can't touch the cars
you're working on). Add to this all the cost cutting measures and
deregulation abuses that have occurred over the past 20 years, and see how
valid a company test is.
Do you have a consistent picture of your subject's thinking about the test
questions in light of what I've said? In his oral responses to the
questions, does he set up a "real-life" answer to the questions. In such a
line of research, I'd ignore the multiple choice answers and focus instead
on whether he gives a solution to the problem posed, although this might be
impossible if the questions are merely signals for the "answers"...that is
if they were not real problems, but rather just "set-ups" for the correct
answers. Or maybe he tried to convert such flat questions into real-life
problems so that he could describe how he would solve them...did you see
any of this?
Are you familiar with Ray McDermott's work with housing project
exterminators in NYC around 1987? They had been performing the job
successfully at low pay, and the Housing Authority wanted to
"professionalize" the jobs at higher pay by using an exam. The choice was
pass the written exam and get a good union job, fail the exam and lose the
job. The union hired Ray's colleague to help the men pass the exam. After
trying a more standard night school approach unsuccessfully (high rates of
test failure), he hired workers who had recently passed the exam to be the
teachersk, and these people managed to increase the pass rates of the test
takers. What these teachers did was to make the connection between the job
and the test questions for their students. I think Ray wrote about this
(he told us this story in a class at UCB), but I never got a good
description about how the process was done...the crucial question: how do
you connect practical experience to test questions about that practice, how
do you create a new (?) kind of thinking.
This is where I get interested because I'm still exploring Vygotsky/Luria's
stuff in Central Asia...the "consciousness thing" Abstract, theoretic
reasoning versus practical, everyday reasoning...scientific thinking versus
spontaneous thinking (although the spontaneous concepts writing was about
children) My hypothesis (following Marx's "being determines
consciousness") is that working class people think differently then middle
class people (different, not inferior...in some cases actually superior!),
and both think differently than the bourgeoisie. This does not mean
capability or potential, just habituation. Of course training (like
schooling) can change the habits, or add new dimensions to the kinds of
thinking used. School tests and discourse are examples of the education
system having the cards stacked against the working class. So far this is
not very original, although in academia such an argument must be qualified
and couched with so much side argumentation that there is no explicit
challenge to the capitalist system. And what doesn't get said is that the
working class reasoning is just as valuable as the middle class style, that
Science, Math, Art, etc can be done without middle class discourse...that
in fact middle class/bourgeois styles are necessary for the defense of
capitalism because they promote reification, commodification, etc (in fact
they historically evolved with capitalist exchange value)
Sorry to go off, but that's why this stuff about consciousness is so
important to me. I'd like to know if you can think about this in light of
your experiences with the Amtrak worker and any others like him.
Well have to go
Write back
Pete Farruggio
>
>This is a true story about "the actual cognitive practices of people in
>realistic contexts, employing the tools available to them for the kinds of
>tasks they actually use". However, it is not a story about an actual
>investigation designed to discover these cognitive practices nor is it a
>projected thought experiment. Instead, it's a story about how three
>different activities purport to represent a single practice and what
>happened to a real individual whose life path was affected by this
>assumption. Where I come in is that I was the teacher to whom this person
>turned to TEACH him how to do behavior x in context C, which he could
>already do in context A and seemed to be able to do, as far as I could
>tell, in context B. Being in this role as a teacher many many times is
>what makes me come to a book like Mike's with great hope and gratitude...
>
>. . . although I can't say it's solved all my problems.
>
>The question here is, was it really always behavior x in all three cases?
>I happen to think not. I'm about ready to say that if you change the
>context, you change the behavior, period.
>
>
>Here is the situation:
>
>
>I had a student, a 35-year old African American man, who was employed by
>Amtrak. He had worked there long enough and was respected by his
>co-workers and was being urged to take a certain test that would result in
>a promotion to carman in the mechanical department. His co-workers and
>boss believed, based on his actual work, that he could pass the test.
>However, every time he took the test, he failed it. Finally they -- his
>boss -- decided that this man had a reading problem and told him to go to a
>community college and take a basic reading class. This is where I met him.
>
>This man was a good student. He seemed to be able to read fine (at least
>better than the typical student in that class) although since he worked the
>night shift he often dozed off in class while sitting over a book or
>papers. Since many kinds of expectations converged around him passing this
>test I tried to figure out in more detail what was going on.
>
>The problem, as he presented it, was that although he could do the work in
>the yard, he could not pass the test about the work. I asked him to bring
>me a copy of the test. He did. It was a 107-item multiple choice test with
>3 pages of diagrams. The copy he brought me happened to have both the
>correct answers marked on it and the wrong answers he had given.
>
>Here is a sample question:
>
> When workers are on, under, or between rolling equipment on a main
>track, which of the following is true?
> a) The equipment must be protected by derails.
> b) A Blue Signal must be displayed at both ends of the equipment to
>be protected.
> c) All switches providing access to the main track must be lined
>away and locked with an effective locking device.
> d) Engines must be uncoupled from cars.
>
>
>The correct answer is b; my student answered c. Obviously, a person could
>get killed by doing this sort of thing wrong. You could also kill someone
>else. You would not want to be working on the same crew as someone who got
>this kind of thing wrong.
>
>I then sat with him and asked him some of the questions and listened to his
>oral answers and compared them with both the correct answers circled on the
>test and the incorrect answers he had given last time he took it. As I
>listened, I realized that there were now, in the world, three kinds of
>evidence about whether he could do this work or not:
>
>1. His actual work in the train yards. Apparently, people who worked with
>him there felt that he did the work well enough to deserve a promotion.
>They were the people who had decided that he had a reading problem.
>
>2. His oral responses to the written questions on the test.These tended to
>be lengthy and discursive, describing the physical environment of the yard
>in great detail. They seemed to flow toward the correct answer; he never
>missed it entirely, but also never hit it perfectly.
>
>3. His written answers on the test. He got 59 out of 107 wrong.
>
>The assumption of the test makers was that the test accurately sampled the
>work and that success at the test would match and predict sucess at work.
>
> First question: is this true? It probably is -- I think we can assume that
>this test has been used enough and its content is critical enough so that
>its validity has been confirmed: people who do well on the test actually do
>know the material. Second question: But can there be other people, for
>whom the test is NOT a valid sampling of the work, could could also do the
>work well and be successful? The recommendation that my student enroll in a
>reading class indicates to me that at least his boss felt that maybe the
>test was not valid for everyone. His boss apparently felt that the test
>ALSO measured reading ability, in addition to knowledge about the work.
>(The boss did not send the student off to study the AAR Field Manual, for
>example, which is cited as one of the sources of information for the test.)
>So maybe the test samples both the work of a carman and ALSO the reading
>level of a carman -- two different cognitive activities, right? Maybe?
>
>Then there is his ability to orally respond to the questions when you're
>sitting face to face with the person asking them, which, I'm going to
>argue, is yet ANOTHER cognitive activity.....
>
>Here is where it seems to me that I am entering a hall of mirrors from
>which there is no escape. How can I know that behavior x is the same in
>all three cases? Why should ANY two different activites be the same,
>anyway? And if they are different, how can teaching one lead to success at
>another?
>
>In real life, people shrug their shoulders and do their best, of course.
>Sometimes a cup of coffee helps, or a walk in the evening when it's cool.
>
>Maybe I can get out of this hall of mirrors by noting that there is
>probably a difference between "analysis of cognitive practice" and
>"cognitive analysis of practice." Is what I'm asking for help in
>"cognitive analysis of a practice" not "analysis of a cognitive practice?"
>Somehow the first seems more do-able. After all, Mike's Chapter 8 is
>titled "The Cognitive Analysis of Behavior in Context" -- and this should
>point us toward the method of analysis rather than analysis of the
>cognitive activity within the practice.
>
>
>Thanks to all for this discussion --
>
>Helena Worthen
>