I think the general observations that have been made are each valid, but
perhaps not always the conclusions drawn from them.
Eugene is certainly right that to assess performance validly you have to
stay as closely within the performance context as possible. This is also
our perspective as researchers, and its outcome is meant to be a
description, and not strictly speaking an evaluation. There _is_ an
important distinction between 'assessment' and 'evaluation' even if the
terms are not consistently used. Assessment tries to mean this description
in rich detail of what's going on; from an assessment you can then go on to
make an evaluation, which judges as better or worse according to value
criteria relevant to some purpose. Of course these value criteria are never
absent from the initial description, just as also in research -- we decide
to pay attention to certain features because they are important to us, and
other dimensions of performance are not described or not in as much detail.
But the values are made explicit in the evaluation, and they change
depending on the purpose of the evaluation. The same descriptive assessment
may be evaluated very differently by different criteria for different
purposes.
I also agree with Eugene that students are inevitably harmed when they are
evaluated by alien criteria, values, and procedures, and for purposes in
which they do not participate. They are then being treated like objects for
someone else's purposes; like tools and cogs in another's machine. Since
they will inevitably be negatively evaluated in ways beyond their control,
they will be hurt. And since these criteria will be neither theirs, nor
directly relevant to their immediate practices and learning, learning
itself will be interfered with and retarded by such evaluations. They are a
form of external and illegitimate interference, at least from the viewpoint
of the student and the learning activity itself.
But it is also true that portfolios, or protocol products of one sort or
another (the Brazilian models here are well-developed ones, and Freire
certainly endorsed such protocol products for mediating learning), CAN be
used in ways that BECOME internal to the immediate learning community. What
goes into them, how they are used, self-evaluation with self-defined
criteria (or negotiated collective, local criteria), using them for group
discussion and reflection .... I don't think Eugene or most of us would
object to this.
The problem comes because the portfolios are not just internal to the local
activity and its direct participant community: they do become also
'boundary objects' that circulate into other communities where they have
very different uses and functions. And when this circulation returns to the
learning community, it can bring with it its harmful and interfering
external effects. These negative effects may be minimized in cases where
the other communities and their purposes are not much at variance with
those developing in the originating community (as in design 'critiques').
They are maximized when the purposes of these other communities are either
antagonistic, or just arrogantly dismissive of the significance of the
local community's (I mean by this term, for example, a classroom, or a
small group of students with or without a teacher) needs, practices, and
emerging values.
But boundary objects are necessary in complex social systems. Something
needs to pass among the different activity settings and local
direct-interaction groups to help coordinate their activities. Especially
in a large scale society where resource flow is conditioned on the movement
of these cross-boundary objects (attendance rosters, grade rosters, budget
requests, budget allocations, regulatory documents, compliance
documentation, transcripts, college and job applications, timesheets,
paychecks, etc.).
The logic of this flow of semiotic artifacts in our society is based on
many ideological assumptions that serve dominant interests and harm
everyone else. In this case the principal one currently, I think, is still
that knowledge is a countable commodity which has intrinsic value. In fact,
value is assigned to various knowledge 'items' in accordance with the
outcome of a struggle or negotiation historically among many interests
(Bourdieu describes this in terms of 'cultural capital'), and even the
item-ization is as much a product of this historical process as is the
ascribed value (cf. Basil Bernstein on strong classification, etc.). But
there are substantial accumulated contradictions in this system of
evaluation, and portfolio assessment is a response to one of the most
critical contradictions of the moment ...
This is the contradiction between official value and use value. The great
idea behind portfolios is not that they are a better way to assess the
canonical commodified-knowledge items of the curriculum, but that they
enable the same portfolio to be evaluated and re-evaluated differently for
different purposes or uses. While every real portfolio may be tailored to
some anticipated evaluator, the portfolio idea is that somebody will like
your portfolio and somebody else will hate it. It ennables a match between
what you can do and what someone else would like to use your labor for. It
by-passes the one-evaluation-fits-all model; you just cannot tell today
from a transcript, or from scores on psychometric tests whether someone
will be a useful employee. The valuations assigned to the test criteria and
methods and biases, or to those that produce grades in traditional
curricula, are mostly worthless for non-academic purposes, they are no
longer good coin for counting cultural capital. It is a very safe
prediction that in the near future traditional grades and scores will count
for very little in gateway decisions compared to evaluator-specific
measures and analysis of applicant-tailored portfolio documentation of
competences. The art of self-presentation will be taught (if not by
schools, then by profiteers), and the content of the portfolio will not
usually be completely generated ad hoc for each evaluator, but will be
selected from a cumulative archive, and then 'dressed' for the particular
occasion.
In the oldest model, powerful interests rigged academic evaluations by
selecting relatively arbitrary content of a non-commodifiable sort (e.g. a
"good" Oxford-Cambridge essay, pre-WW2), with the result that educational
preferment was almost entirely class-habitus based -- except for engineers,
where competence mattered more than class and style, and who began the
democratic commodification model. When that model was extended historically
to subjects where competence remained more a matter of taste than of
results (and this is still a great debate in higher education), there were
bitter (and continuing) struggles over the style vs. test-score model, and
within the test-score model over 'curriculum' (i.e. what gets most points
on the test). But as the social division of labor has proceeded to get more
and more technical, there are now more and more kinds of 'results' and
perhaps also kinds of 'style' that must be judged, and NO single test (or
curriculum) can serve the diversity of potential criteria of evaluation
that may be applied. In some sense the fall of the 'transfer of training'
or 'general intelligence' model is also a product of this historical
specialization and complexification of society -- it just won't work
anymore to pretend that because you did well on an IQ test, or any kind of
test, you will actually be of any use in a genuinely productive activity.
In a way, then, I am arguing for a kind of democratic progress in the move
from holistic essays to item-by-item testing to portfolio assessment. Of
course there has been no corresponding progress in _evaluation_; it is
still done by those who control the resources and it still inevitably harms
most of us. But so long as we maintain an ecosocial order that is a complex
as ours is (differentiated by specialized activities which then need to be
coordinated), assessment boundary objects will need to flow from schools to
evaluators, and the evaluations will flow back and do harm -- but also in
some limited sense good: some people will get access to greater resources.
Unlike art, most school learning is not undertaken as an activity for its
own sake; it is undertaken as preparation for participation in other
activities -- and assessment boundary objects must link the schooling
activity contexts with those for which schooling prepares. De-linking them
is tempting, but dangerous.
I know that for students, esp. very young students, school activities can
become valuable in their own right. For all students, some of the time. But
here we also meet a contradiction, well known to educators: to the extent
that school encourages students to engage in learning for its own sake,
which is the condition under which most people learn best, it risks
students educating themselves in ways that will not be adapted to the other
activity settings in our society -- about which students know next to
nothing and are taught next to nothing -- where school learning is expected
to become useful in the sense of economically productive.
I think sometimes we forget that there ARE contradictions built into our
social system; that it is NOT possible to satisfy all its demands ... which
is indeed why, slower or faster, it continues to change.
JAY.
---------------------------
JAY L. LEMKE
PROFESSOR OF EDUCATION
CITY UNIVERSITY OF NEW YORK
JLLBC who-is-at CUNYVM.CUNY.EDU
<http://academic.brooklyn.cuny.edu/education/jlemke/index.htm>
---------------------------