Jay, I certainty agree with all what you said. I have only one small
comment coming from your statement:
>In a way, then, I am arguing for a kind of democratic progress in the move
>from holistic essays to item-by-item testing to portfolio assessment.
I just what to have a further movement and not a stop at portfolio
assessment... :-)
I really like that portfolio assessment highlight its subjectivity and lack
of reliability (i.e., negotiation character). With grades and tests this
feature of divorced assessments is less noticeable. I like to push this
"crisis" of the assessment reliability even further.
Take care and thanks for your very thoughtful and helpful (as usual!)
elaboration.
Eugene
-----Original Message-----
From: Jay Lemke <jllbc who-is-at cunyvm.cuny.edu>
To: XMCA LISTGROUP <xmca who-is-at weber.ucsd.edu>
Date: Sunday, February 28, 1999 11:25 PM
Subject: boundary objects and evaluation, aka portfolios
>Coming back to a long list of xmca messages after a busy week or so, I read
>with interest the various points made about assessment, evaluation, and
>portfolios pro and con.
>
>I think the general observations that have been made are each valid, but
>perhaps not always the conclusions drawn from them.
>
>Eugene is certainly right that to assess performance validly you have to
>stay as closely within the performance context as possible. This is also
>our perspective as researchers, and its outcome is meant to be a
>description, and not strictly speaking an evaluation. There _is_ an
>important distinction between 'assessment' and 'evaluation' even if the
>terms are not consistently used. Assessment tries to mean this description
>in rich detail of what's going on; from an assessment you can then go on to
>make an evaluation, which judges as better or worse according to value
>criteria relevant to some purpose. Of course these value criteria are never
>absent from the initial description, just as also in research -- we decide
>to pay attention to certain features because they are important to us, and
>other dimensions of performance are not described or not in as much detail.
>But the values are made explicit in the evaluation, and they change
>depending on the purpose of the evaluation. The same descriptive assessment
>may be evaluated very differently by different criteria for different
>purposes.
>
>I also agree with Eugene that students are inevitably harmed when they are
>evaluated by alien criteria, values, and procedures, and for purposes in
>which they do not participate. They are then being treated like objects for
>someone else's purposes; like tools and cogs in another's machine. Since
>they will inevitably be negatively evaluated in ways beyond their control,
>they will be hurt. And since these criteria will be neither theirs, nor
>directly relevant to their immediate practices and learning, learning
>itself will be interfered with and retarded by such evaluations. They are a
>form of external and illegitimate interference, at least from the viewpoint
>of the student and the learning activity itself.
>
>
>But it is also true that portfolios, or protocol products of one sort or
>another (the Brazilian models here are well-developed ones, and Freire
>certainly endorsed such protocol products for mediating learning), CAN be
>used in ways that BECOME internal to the immediate learning community. What
>goes into them, how they are used, self-evaluation with self-defined
>criteria (or negotiated collective, local criteria), using them for group
>discussion and reflection .... I don't think Eugene or most of us would
>object to this.
>
>The problem comes because the portfolios are not just internal to the local
>activity and its direct participant community: they do become also
>'boundary objects' that circulate into other communities where they have
>very different uses and functions. And when this circulation returns to the
>learning community, it can bring with it its harmful and interfering
>external effects. These negative effects may be minimized in cases where
>the other communities and their purposes are not much at variance with
>those developing in the originating community (as in design 'critiques').
>They are maximized when the purposes of these other communities are either
>antagonistic, or just arrogantly dismissive of the significance of the
>local community's (I mean by this term, for example, a classroom, or a
>small group of students with or without a teacher) needs, practices, and
>emerging values.
>
>But boundary objects are necessary in complex social systems. Something
>needs to pass among the different activity settings and local
>direct-interaction groups to help coordinate their activities. Especially
>in a large scale society where resource flow is conditioned on the movement
>of these cross-boundary objects (attendance rosters, grade rosters, budget
>requests, budget allocations, regulatory documents, compliance
>documentation, transcripts, college and job applications, timesheets,
>paychecks, etc.).
>
>
>The logic of this flow of semiotic artifacts in our society is based on
>many ideological assumptions that serve dominant interests and harm
>everyone else. In this case the principal one currently, I think, is still
>that knowledge is a countable commodity which has intrinsic value. In fact,
>value is assigned to various knowledge 'items' in accordance with the
>outcome of a struggle or negotiation historically among many interests
>(Bourdieu describes this in terms of 'cultural capital'), and even the
>item-ization is as much a product of this historical process as is the
>ascribed value (cf. Basil Bernstein on strong classification, etc.). But
>there are substantial accumulated contradictions in this system of
>evaluation, and portfolio assessment is a response to one of the most
>critical contradictions of the moment ...
>
>This is the contradiction between official value and use value. The great
>idea behind portfolios is not that they are a better way to assess the
>canonical commodified-knowledge items of the curriculum, but that they
>enable the same portfolio to be evaluated and re-evaluated differently for
>different purposes or uses. While every real portfolio may be tailored to
>some anticipated evaluator, the portfolio idea is that somebody will like
>your portfolio and somebody else will hate it. It ennables a match between
>what you can do and what someone else would like to use your labor for. It
>by-passes the one-evaluation-fits-all model; you just cannot tell today
>from a transcript, or from scores on psychometric tests whether someone
>will be a useful employee. The valuations assigned to the test criteria and
>methods and biases, or to those that produce grades in traditional
>curricula, are mostly worthless for non-academic purposes, they are no
>longer good coin for counting cultural capital. It is a very safe
>prediction that in the near future traditional grades and scores will count
>for very little in gateway decisions compared to evaluator-specific
>measures and analysis of applicant-tailored portfolio documentation of
>competences. The art of self-presentation will be taught (if not by
>schools, then by profiteers), and the content of the portfolio will not
>usually be completely generated ad hoc for each evaluator, but will be
>selected from a cumulative archive, and then 'dressed' for the particular
>occasion.
>
>
>In the oldest model, powerful interests rigged academic evaluations by
>selecting relatively arbitrary content of a non-commodifiable sort (e.g. a
>"good" Oxford-Cambridge essay, pre-WW2), with the result that educational
>preferment was almost entirely class-habitus based -- except for engineers,
>where competence mattered more than class and style, and who began the
>democratic commodification model. When that model was extended historically
>to subjects where competence remained more a matter of taste than of
>results (and this is still a great debate in higher education), there were
>bitter (and continuing) struggles over the style vs. test-score model, and
>within the test-score model over 'curriculum' (i.e. what gets most points
>on the test). But as the social division of labor has proceeded to get more
>and more technical, there are now more and more kinds of 'results' and
>perhaps also kinds of 'style' that must be judged, and NO single test (or
>curriculum) can serve the diversity of potential criteria of evaluation
>that may be applied. In some sense the fall of the 'transfer of training'
>or 'general intelligence' model is also a product of this historical
>specialization and complexification of society -- it just won't work
>anymore to pretend that because you did well on an IQ test, or any kind of
>test, you will actually be of any use in a genuinely productive activity.
>
>In a way, then, I am arguing for a kind of democratic progress in the move
>from holistic essays to item-by-item testing to portfolio assessment. Of
>course there has been no corresponding progress in _evaluation_; it is
>still done by those who control the resources and it still inevitably harms
>most of us. But so long as we maintain an ecosocial order that is a complex
>as ours is (differentiated by specialized activities which then need to be
>coordinated), assessment boundary objects will need to flow from schools to
>evaluators, and the evaluations will flow back and do harm -- but also in
>some limited sense good: some people will get access to greater resources.
>Unlike art, most school learning is not undertaken as an activity for its
>own sake; it is undertaken as preparation for participation in other
>activities -- and assessment boundary objects must link the schooling
>activity contexts with those for which schooling prepares. De-linking them
>is tempting, but dangerous.
>
>I know that for students, esp. very young students, school activities can
>become valuable in their own right. For all students, some of the time. But
>here we also meet a contradiction, well known to educators: to the extent
>that school encourages students to engage in learning for its own sake,
>which is the condition under which most people learn best, it risks
>students educating themselves in ways that will not be adapted to the other
>activity settings in our society -- about which students know next to
>nothing and are taught next to nothing -- where school learning is expected
>to become useful in the sense of economically productive.
>
>I think sometimes we forget that there ARE contradictions built into our
>social system; that it is NOT possible to satisfy all its demands ... which
>is indeed why, slower or faster, it continues to change.
>
>JAY.
>
>
>
>---------------------------
>JAY L. LEMKE
>PROFESSOR OF EDUCATION
>CITY UNIVERSITY OF NEW YORK
>JLLBC who-is-at CUNYVM.CUNY.EDU
><http://academic.brooklyn.cuny.edu/education/jlemke/index.htm>
>---------------------------