Re: [xmca] Automating Dynamic Assessment (DA)?

[David Kellogg <vaughndogblack@yahoo.com>]

A couple of years ago I was at a conference on language testing here in Seoul where the new Computer Based Tests (mostly TOEFL and TOEIC) brought out by Educational Testing Services were being flogged (in both senses of word, that is, touted on the one hand and tortured on the other).
 
I was in the torturing camp. It seemed to me at the time that there was an irredeemable flaw to the tests. You get a test item. If you get it right, the machine gives you a more difficult one. If you get it wrong, the machine gives you an easier one. In this way the machine is supposed to "home in" on something called your "true ability" or your "true level" or whatever.
 
Let us leave aside for a moment the question of whether there is something called "true ability" or "true level" in language use (you have to check that particular piece at the door when you enter a language testing conference). The "homing in" idea (which is clearly central to Mike's problem of automating DA too) means that the first problems you have on the test are worth WAY more than the subsequent problems, and the last problems you have on the exam are practically worthless. 
 
You see the flaw. The first problems on the exam are precisely the ones that are assigned at random, because the machine has no idea where to place you. And those are the problems that decide your fate. Now, when I raised this in the discussion I was told that this was, in fact, a very difficult and complex problem, but that it had been solved by ETS. However, we were told, the solution was copyright and could not be shared at the conference.
 
Well, I think I can tell everybody on the list, for free, the "solution" that ETS came up with. It was simply to ignore the problem, jettison the whole idea of item independence and with it the reliability of the examination. (Interestingly, CBTs are not available here in Korea for somewhat different reasons: Korean students are dynamic testees, they tend to share the questions on the internet after taking the tests, and it turns out that the item bank used by ETS is pitifully small....)
 
Obviously, TOEIC/TOEFL are not in any sense dynamic tests. But the problems of animation remain the same. Paula, who is now (as I understand it) building an automated version of the Chapter Five blocks test, has to deal with the question of which blocks to reveal to the children first or whether to let them choose at random. With the blocks test, the problem is not too serious, since all of the blocks are equally unfamiliar. But this is NEVER true of a linguistic item.
 
I think that Poehner and Lantolf have really bitten the bullet with this latest article. They have decided that the assumption of test VALIDITY (that is, the idea that there is an underlying proficiency which is stable and testable) has to go. The assumption in DA is that there is NO underlying proficiency, or if there is, it is NOT stable, or if it is stable than the point of the test is not to assess it but to DESTABILIZE it. Ergo, no DA can have construct validity in the classical psychometric testing sense. That's the point Johnson-Laird is making.
 
But contrary to what Johnson-Laird says, there is a way in which you can partially "model" on a computer the developmental stages of a child laid out in Volume Five of the collected works at least as well as you can "model" quantum mechanics or the economy (that's a VERY bad example, but it's not mine!). The way is to assume that the various RELATIVELY STABLE stages of development (birth, infancy, early childhood) are more or less linear accretions to a particular neoformation (instinctive life, imitation, speech). Here is where we are going to find learning.
 
What cannot be modeled are the crises. Now, the truth is that there are lots of things that cannot be modeled on a computer in physics (e.g. the actual shock wave created by a supersonic plane, or by the earth in the solar wind, as opposed to what precedes it and what goes after it). And just about EVERYTHING that means ANYTHING in an economy cannot be "modeled" on a computer in a meaningful way (else we would not be in our present pickle!). All these unmodelables are critical phenomena in more or less this way. But it's precisely here that we are going to find development.
 
I agree wholeheartedly with Poehner and Lantolf when they jettison the "assessment" component of DA and simply concentrate on dynamic learning. The problem I see is that they have also jettisoned development: development has become redefined as nothing more than the product of learning, that is, the ability to do something on your own that you once could do only with assistance. 
 
But that's not what development is. There are some forms of learning which are really, critically, qualitatively developmental (hint: after age two, they almost ALL have to do with language, and many aspects of foreign language learning are certainly among them). But as Vygotsky says, there are also many many things we learn that are developmentally inert (he has a thing about typing, playing golf, and riding a bicycle, but he also mentions Thorndike's tests of estimating the length of line segments and then trying to transfer to the size of angles: both of these are considered nondevelopmental for school age populations). 
 
It doesn't matter whether you do these things alone or with assistance; they do not help elementary school chldren develop in any important way. We cannot say, for example, that a child who rides a bicycle is more developed than a child who rides a unicycle or even than a child who cannot ride a bicycle, and we cannot say that a boy who grows up in England playing cricket is somehow more developed than one who does not. 
 
The same thing cannot be said about literacy, or for that matter foreign language learning: a child who knows how to do these things can still do everything that the child who cannot do them can do, and do them in a different way, a more conscious, deliberate and free way, and a whole lot more. That is why we can say that such a child is qualitatively more developed than an illiterate or a monolingual child.
 
Now I used to think that the distinction between learning-that-leads-development and learning that does not could be made at the level of the curriculum: that is, typing is always developmentally inert knowledge, and foreign language learning is always revolutionary and developmental. I still think that there are certain subjects that need to be on the curriculum for developmental reasons, and foreign language learning is one of them. But I also think that the distinction between learning-that-leads-development and inert learning exists within subjects as well as between them, and there is a great deal of what we do in a classroom that is developmentally inert for the kids we are teaching. 
 
I think what is really exciting about DA is not that it tests the kids, but that for the first time it allows us to put the subject itself to the test, and find out which kinds of knowledge are radically restructuring. That, by the way, is the real rationale for what Paula is doing these days. In my opinion.
 
David Kellogg
Seoul National University of Education 
 
-- On Sat, 11/27/10, mike cole <lchcmike@gmail.com> wrote:


From: mike cole <lchcmike@gmail.com>
Subject: [xmca] Automating Dynamic Assessment (DA)?
To: "eXtended Mind, Culture,Activity" <xmca@weber.ucsd.edu>
Date: Saturday, November 27, 2010, 10:49 AM


Our recent discussions have pushed me to revisiting interesting texts that
appeared in and around the
first socio-cultural studies conference which served as the setting for me
the remarks I sent out a few days
ago.

This revisiting turned up Harry Daniels' (1990) very useful Introduction to
Vygotsky which has an interesting
introduction where some of the issues frazzling around here are dealt with
in an interesting way. Several of the
issues that it raises seem relevant to xmca discussions. The one I have
picked out here is the question of the extent
to which computer technology has achieved sufficient power to model complex,
non-linear, open systems, such as
those characteristic of the kind of sophisticated pedagogical dialogues
illustrated in the P&L text.  Harry points us
to Phil John-Laird's (1986) critique of Vygotsian ideas as summarized by Jim
Wertsch in his 1985 book. I had never read it,
but was intrigued to read:

Modern Vygotskians must come to terms with the impact of computation on
conceptions of mind, They must offer an explicit theory
that can be modeled in a computer program in the same way that one can
model, say, the economy, or the weather, or quantum mechanics.
No Marxist psychology is likely to meet that demand, and Vygotsky's grand
theory will probably not be followed by another in the foreseeable
future..... Vygotsky was an artist trying to construct a scientific
psychology in an era when the only language for theories was the vernacular.

(!!!)

A whole lot of stuff one could engage with in all of that, but what strikes
me as how well Johnson-Laird's perspective, which can fairly be said to
have some contemporary influence, provides a perfect reason to find attempts
to model DA interesting. What, if instead of being replicas of reality,
AI theories of human action were treated as "just another" tool for
accomplishing the goal of merging instruction and assessment in an effective
activity that is part of a formal school regime? Can highly scripted
interactions that none-the-less require ongoing imaginative work by
participants be modeled to a "sufficient degree" to pass muster?  It seems
to me like work we could learn from. Locally there is widespread interest in
artificial social interaction -- although I have not heard the word societal
whispered about.

Could DA be "automated"? If it could be automated, what would it tell us
about the society that used it? Would it be as rigid as getting help when
contacting a government agency by telephone? Would it increase efficiency,
or strengthen the bars on the iron cage?

mike
__________________________________________
_____
xmca mailing list
xmca@weber.ucsd.edu
http://dss.ucsd.edu/mailman/listinfo/xmca




__________________________________________
_____
xmca mailing list
xmca@weber.ucsd.edu
http://dss.ucsd.edu/mailman/listinfo/xmca