Sustaining Model Systems of Educational Activity:

(Paper Presented at Symposium Honoring the Work of Ann Brown,

Berkeley, California, January 19-20, 2001)

Michael Cole

Laboratory of Comparative Human Cognition

U.C.San Diego

Once upon a time, in a far away ivory tower by the East River, my colleagues and I were engaged in what, at the time, was considered a peculiarly odd enterprise. We had come to distrust the power of experimental psychology, as we had learned it at our mentors’ elbows, for illuminating important aspects of cognitive development in a manner that could be validly generalized beyond their artificially produced conditions of existence. This distrust had arisen from our experience conducting research in markedly different societies, especially societies whose cultures did not include heavy dependence on modern technologies of transportation, communication, and industrial production, regular practice in the arts of reading and writing, or experience in such institutions as formal schooling or government bureaucracies.

Briefly summarized, standard cross-cultural research invited the conclusion that without the benefits of literacy, numeracy, and extensive schooling, mental development is stunted. This retardation could be regularly illustrated by the use of standardized tests or standardized experimental procedures that measure speed and complexity of learning under a variety of well defined, and highly controlled, widely used experimental conditions (Cole, Gay, Glick, & Sharp, 1971). However, this conclusion was suspect because individuals who appeared retarded when administered even carefully crafted tasks appeared to display the presumably absent cognitive processes when task procedures were radically changed. The general strategy in such changes was to approximate learning situations which were closer to their everyday lives. Suspicion was also aroused when tasks that Liberian farmers found relatively easy were re-imported to the U.S. where their presumably superior counterparts were now cast in an unflattering comparative light.

After we had been engaged in such work for some time, work that involved us in ethnographic and linguistic analyses of local settings and materials as a means for redesigning experiments we reached the now-obvious conclusion that the standard tasks of experimental psychology have their historical origins in the institution of schooling in modern Europe and the United States, and hence, there is an incestuous and distorting relationship between traditional experimental procedures and historically recent social practices that are frequent in our own society.

Consequently, we began a search, using a motley array of tools, many of them improvised, others borrowed from researchers with different goals in mind, in the hope of understanding how performance in psychological tests relates to performance in a wide variety of less, or differently, constrained tasks of the kind that people encounter routinely in a wide variety of circumstances in their everyday lives. We also searched for theories that could help to guide this effort. In technical parlance, we set out to investigate the ecological validity of cognitive, experimental tasks.

During this period, in 1977, Ann and Joe came to visit us. I do not recall the exact reason for their visit. It may have been because we, like them, were conducting some training studies with children who were generally performing poorly in their school to see if short term training and familiarity with the semantic organization of the materials that made up the core curriculum could induce generalized use of strategies. Or it could have been that they had encountered one of our early reports indicating that remembering as it occurred in the not-so-school-like setting of an afternoon cooking or nature club doesn’t appear to have much to do with the kinds of remembering required either in standard memory training tasks or school assignments.

One set of observations, in particular, formed a link to our later interactions. Early in the course of my lab’s work comparing the behavior of children in their classrooms, individualized testing, and our afterschool clubs, we discovered that one of the children who we considered generally competent and helpful afterschool was diagnosed as "learning disabled" by his teacher and the school psychologist. If we had begun to doubt our power as cognitive analysts up until then, it was nothing compared to the challenge posed by learning that we had been working with a child who could not read in his classroom, but appeared perfectly competent when dealing with recipes and guides to growing beans indoors during the New York winter. The title of the never-published monograph we wrote about this work, "Ecological invalidity as an axiom of cognitive psychology" indicated the dour conclusions to which we were drawn.

At about the time this work was completed, our Laboratory moved from the ivory tower by the East River to an adobe tower on a hill overlooking the Pacific Ocean and we decided that the question of the ecological circumstances associated with the notion of a "learning disability " needed to be followed up. Not only did we want to go beyond our single case study but to follow children into a variety of theoretically selected cultural contexts including not only their regular classrooms, but their pull-out special education sessions, their interactions on the play ground, and their activities at home and in their neighborhoods. We also wanted to experiment with various kinds of diagnostic and strategy-training techniques that Ann and Joe were then investigating to see how this population of children might, or might not, be helped by them.

The first year of this plan began smoothly enough. We found schools that agreed to participate, we identified populations of children to whom we gave batteries of tests that were routinely used to identify children fiting the legal definition of learning disabled, and we began to make classroom observations. We even began piloting alternative diagnostic procedures and strategy training tasks that Ann and Joe had designed. We included real learning curves in our yearly progress report!

But trouble arose right at the start of the second year when we set out to implement the full research plan. The classroom teacher who was most enthusiastic about working with us was assigned to a new school. Other teachers said it would be alright to observe and record the children during special pull-out sessions, but not during regular instruction. Our access to the resource room was restricted by a jurisdictional dispute involving the school district and the special education teacher.

Faced with these difficulties, we shifted our focus to another school which we had planned to use in a later phase of the research, and for which we had collected baseline data. But no sooner had we begun our work in this new school than the teachers who had agreed to participate in the research asked to withdraw. Over the summer the school district had mandated a very strictly regulated curriculum with minute-by-minute scheduling of activities. As one teacher told us, " If you have materials that I can put in the room that will guarantee the children will learn effectively, you are welcome. Otherwise you are not."

We could not, of course, make such a guarantee. So at this point we faced the prospect either of giving up the work and dismissing our research team or finding a radically new way to pursue our research goals. After an emotional meeting with the principal and teachers, we decided to move the entire operation to the after school hours and take responsibility for instruction ourselves. Thirty parents of children we had identified as experiencing chronic learning difficulties agreed to have their children participate in our program for two hours a week. The program ran on Monday and Wednesday or Tuesday and Thursday for the next two school years with strong support from school staff.

This shift in strategy meant that we had to abandon the luxurious role of the observing researcher and put together a curriculum that we, ourselves, could run. And we had to do it in a hurry. It was an impossibly difficult task that was accomplished largely thanks to the brilliant and tireless efforts of Peg Griffin, whose extensive background in reading research allowed her quickly to construct four different reading activities. Each activity was designed for small group instruction and each was organized on the fundamental notion that reading is a process of comprehending the world mediated by print. These four reading activities, one of which was a modification of the reciprocal reading procedures that Anne Marie Palincsar and Ann were developing at that time, formed an essential core of our afterschool educational system which, at the suggestion of the school principal, we called Field College.

In addition to the four small group reading activities, we included two other major components — individualized cognitive strategy training based on Ann and Joe’s work, and a hybrid, game-like set of computer based activities focused on reading, strategy training, and elementary arithmetic that we called "The Fifth Dimension." We also maintained our commitment to the study of children in various contexts by observing them in their everyday classrooms and paying frequent visits to their homes and neighborhoods. Very importantly, we linked these activities to a research methods course at the University; undergraduate researcher/students became, and remain, a key element in the educational activities we design.

It is not my intention here to re-cap all the work we conducted at Field College, much of which has been published in other sources (See Cole, 1996; LCHC, 1982). Suffice it to say that the overall results were satisfying on many grounds. Our experimental reading tasks appeared effective both as a means for diagnosing individual children’s barriers to reading acquisition while simultaneously proving to be effect media of instruction. The children attended regularly, their behavior in school improved, first their social behavior, and then their academic performance. As researchers we learned an enormous amount about what it means to engage in design research, as that term has subsequently come to be used.

As but a single example of what it means simultaneously to invent new teaching/learning activities, implement them, analyze them in theoretically relevant terms, and then use the results to modify instructional design for a next day’s lesson, I have selected a here-to-for unpublished fragment from the second year report of the project. This fragment has the added virtue of providing an excellent example of the spirit with which Ann and Joe entered into what was a risky and difficult enterprise.

On the day in question, the trained reading teacher who was supposed to run the lesson ("Ms Griffin") was absent. Ms. Griffin had set up a lesson that she thought could be run by Cole: to reconstruct a previously read text from sentence fragments. Brown and Campione were present as observers. As materials for the lesson, a few passages of text modeled on a text that the students had been reading previously were cut into strips. Each strip contained a sentence or a sentence fragment, typed in bold print. The children were each given an envelope with the entire story cut into 23 strips. The text from which the story had been made up was nearby in case the children felt that they needed to refer back to it. Glue sticks in hand, interpersonal issues organized into abeyance, the children began the task of reconstructing the text.

Twenty minutes had been allotted for this activity, but it didn't take long for Cole to figure out that twenty minutes was not going to be enough time. After a few minutes of struggle, things weren't going too well. Little reading was occurring. Some of the children were playing lay-out artist; they were busy arranging strips of paper in a newspaper format but paying no attention to the semantic content of the strips. Some of the children had pasted a single phrase or sentence on their sheet of paper, but the remaining paper strips lay in jumbled heaps around the table. Some time was spent in trying to regularize each child's stock of phrase strips, but by the time one child got organized, another was dropping out of the process in confusion and its ensuing state, boredom. The noise level was slowly rising.

Cole, seeing that things were going poorly, reached for the original text from which the phrases had been drawn. As he did so, Brown and Campione, seeing that the lesson was in immanent danger of collapse, moved up to the table, elbowed their way between distracted children, and started to puzzle over the tangle of paper and print.

Cole began to read the story slowly, commenting from time to time on the relationship between what he was reading and the phrases with which the children were working. He found some important landmarks (for example, that discussion of "difference machines" precedes discussion of "analytic machines") which he pointed out. Brown and Campione played cooperating chorus. They emphasized the significance of benchmark phrases, suggesting that one of the strips clearly came last.

Everyone set to work again. For a while, several of the children worked with Cole, Brown and Campione in an attempt to come up with "at least one good story to show to Ms. Griffin, the reading teacher." Progress was made, so that after 20 minutes or so of discussion, trial and error work, and lots of phrases saved from a gluey grave, parts of the story could be found on a few sheets of paper. But time was up, so the adults called a halt to the work with the promise that the kids would all get another chance.

The entire interaction was unusual for what we might call its affective tone. At the outset the tone was decidedly negative. The children vocally agreed that the task was dumb, uninteresting, and an impediment to playing on the computers beckoning from across the room. However, the task was potentially engaging. The goal was clear enough, but it looked very tough to achieve, and once failure loomed, the children started to reject it. The critical juncture was when Cole picked up the text, clearly intent on success in the face of adversity, and the Brown-Campione duo moved in to help. It was clear that the adults did not know precisely what to do, but also that they were willing (and apparently happy) to tackle the job. As the adults began to chat with each other about possible sentence-phrase combinations, the children started to chime in with relevant suggestions. There were enough plausible suggestions and confusion so that everyone felt a part of the activity. Expertise was not clear cut. But everyone was reading. Almost all of the children read strips of various kinds repeatedly, none of the adults could piece all the strips where they belonged. Adults and children had entered into a task together, together accepted and worked toward a goal. When time was up, everyone was satisfied that a real piece of work had been done, even though they had "failed" in the narrow sense. The triumph consisted in getting a differentiation of the senses of failure. Everyone succeeded in reading for meaning.

The following day, Ms. Griffin repeated the lesson with another group of children. Forewarned, she prepared sheets on which some of the strips had been pasted ahead of time, so that the task of arranging all the strips was greatly simplified. Now it was necessary only to arrange the missing strips. Several milestones in the text were placed on the sheets as an additional aide to finding the missing pieces.

In these circumstances, Ms. Griffin could handle the task alone with the children; social resources for engendering educational activity were converted into material, written resources. This theoretically guided conversion of "social" into "cognitive" resources was supported by the way in which the interactions between Ms. Griffin and the children was structured. All of the children worked like beavers on their own story "puzzles." Ms. Griffin moved from one child to the next, responding to calls for help. She "failed" only in the sense that the children's requests for assistance to carry out their task were coming faster and more furiously than she could cope with smoothly. With the appropriate structuring of the story making task, these children took over the activity so thoroughly that Ms. Griffin became an amuensis instead of a guide. She cheered their suggestions, wrinkled her nose at strange combinations which threatened to stick to the paper in strange places, and generally did her best to indicate her pleasure at being able to participate in the children's activity. Once again children gained a lot of practice in learning to read, and the potential of the initial activity, in its redesigned form, was realized.

Some interpretive remarks

In thinking about the changes in our children's behavior that were produced by the two different versions of a "writing" assignment, it appeared that in each case the children shifted from a situation where there was a teacher and students to one in which the students "got into" the task, taking it as their own, and thereby engaging in precisely the kind of learning activity we were seeking to engender. This shift coincided with an ability on the children's part to engage in interactions with the adults in which a division of labor had been arranged whereby each party to the interaction, adult and child, contributed to the utmost of their abilities given the constraints provided by the text and the entering skills of all the parties. Adults were no longer "holding out" on the children; they were contributing at their highest level of understanding. They displayed this understanding, which was not the final product, but useful actions that presupposed the overall goal (read these darn slips of paper and figure out what the story must be). In moving to a system controlled by a "known answer machine" to a "knows something about how to find the answer" person, we had recreated an essential feature of natural environments for language acquisition but just as natural environments for learning-to-read-as-a-problem-solving-tool

It is interesting to note that there were no plans ahead of time to have Cole teach a lesson that was too difficult; there were no plans for Brown and Campione to move into the center of the lesson to help out; Griffin could not foresee her role as consultant/kibbitzer once the task had been more fully structured. These roles grew "spontaneously" in the interaction. "Spontaneous" in this case means that interactional dynamics made the form of interaction seem natural and necessary, "given the conditions." The conditions were the base necessity to keep the children in the interaction in the absence of strong negative sanctions. Either we made the activity into something that the children would make their own, or we failed. In making the repairs necessary to keep the children in the interaction, we hit upon a system that got them to enter into it in the way that Vasilli Davidov used to call educational activity. He liked to say that educational activity would never happen in a school because children never are allowed to discover educational goals there. He was almost right.

Ann ends her 1992 paper on design experiments (Brown, 1992) with a number of cautionary tales about educational innovations such as Field College, although she oriented her comments to reform of in-school activities. The following cautions are most germane to the work I am about to describe:

Our example certainly is ripe for criticism as a romanticized anecdote, although as part of a larger corpus of videotaped events in which similar dynamics were observed, we could make a good case that such events were a routine occurrence in Field College. The major value of such descriptions has been to provide data about the processes of interaction which could plausibly be argued to give rise to the children’s improved performance in their regular classrooms and on our specially designed tests. The example also provides a clear example of the tension between designing effective lessons through in-situ modifications of pre-existing designs and the need for evaluation to which Ann refers. What the example cannot provide, however, and what Field College could not provide, was a specification of the core set of features which made it successful. Was it our specially designed reading procedures? The strategy training sessions? The computer based activities? The opportunity to interact with undergraduates? Could any of this be replicated, let alone sustained or disseminated?

Ann was not optimistic in her prognosis for the widespread dissemination of the communities of learning approach she believed in, because she recognized that it would be profoundly disruptive of "practice as usual." She also knew that to carry out the research needed to assess widespread dissemination would require mastery of new fields of knowledge, such as the sociology of dissemination. And, though she didn’t mention it explicitly, such work would require time…. A lot of time. And time was something she didn’t have in sufficient amounts.

Well, one of the things friends are for is to carry on valued tasks that have gone unfinished. In my case, it means carrying on the tasks of creating a methodology that is adequate to the social science study of educational design innovations, to do so in a way that does not rest entirely on engaging anecdotes (without reducing the processes of learning and development to a graph or a table of numbers) and to study the processes by which such innovations can be disseminated and sustained. As luck would have it, this is not a task that I discovered as a consequence of Ann’s death, or even as a consequence of her work on design experiments. Rather, it is a task that my colleagues and I began in 1986, in response to the inadequacies of educational research which we discovered in company with Ann and Joe, but pursued in an independent fashion, a somewhat different socio-ecological niche, and with a somewhat different, but ultimately compatible goal in mind.

As I have written elsewhere (Cole, 1996), in 1984, while we were wondering if there was life after Field College, LCHC took on the task of writing a study for the National Research Council about the barriers to inclusion of minorities and women in technological and scientific education (Cole, Griffin, & LCHC, 1987). As our multi-disciplinary team was busy gathering data to address this question, I received a report from the AAAS about the characteristics of successful programs for carrying out the mission we had set out to address (AAAS, 1984). The need, it seemed, was not for research to figure out how to create successful programs, the problem was to figure out how to sustain successful programs

In a certain sense, the answer to the question of why successful innovations fail and have constantly to be reinvented was already well known: institutions welcome innovations so long as they are compatible with institutional goals and are supported by external funds. But the host institutions do not integrate the innovations into their core activities, so when the extra money goes, so does the innovation. Still, considering the importance of the issues involved (we were being told at the time that we were a nation at risk owing to our poor educational performance, rhetoric which has not changed much in the intervening decade and a half) we were interested in how decent people could allow such failure to replicate itself with such regularity in contradiction to their sincerely held personal values. How did the process work, up close and personal? We decided to address this question directly, using the computer-based activities developed at Field College, The Fifth Dimension, as our candidate "successful innovation."

Our plan was quite simple, not to say, "simple minded." We would initiate Fifth Dimensions in a variety of institutions and study the dynamics of their change, and presumed demise, over the period beginning with their initiation through to the end of their initial funding. We would not rely on either single anecdotes or single cases. Instead, we would elaborate quantitative measures to monitor the relative success of the innovation on the children and qualitative measures to index the dynamics of change in several institutions simultaneously. We would pay special attention to the period of transition, when regular funding came to an end, but we would be alive to the possibility of crises developing at any stage of the processes.

As Ann noted in her 1992 paper, it is important to develop and articulate a set of core principles which any educational activity must embody in order to be considered a successful replication. This same requirement, of course applies to evaluating the sustainability of an innovation. An educational innovation can "die from the inside" even as it continues to operate in an inappropriate fashion from the perspective of its initiators.

One of the interesting things that I have learned in 15 years of studying sustainability of the 5^th Dimension is that it is almost certainly unrealistic to believe that it is possible to specify of all the core principles ahead of time, or even on the basis of a single such system. The reason for this difficulty follows from the fact, noted by Allan Collins (1999, p. 291), that "design experiments have many dependent variables that matter, although the experimenter may not pay attention to them all them all." I would go further and claim that design experiments involving educational activity are complexly constructed social systems in which it is simply not possible to be sure at all times what combination of factors is at work to produce the phenotypical appearances. All such systems are emergent products not only of factors identified as internal to the system, but factors that involve the necessary openness of such systems to the social systems in which they are embedded.

However, it is important that researchers continuously attempt to formulate whatever principles appear to be essential to the operation of the system, expanding and modifying the set as they go along. In 1997, after I had been conducting research on the 5^th Dimension for 10 years, I produced the following set of core principles in collaboration with Bill Blanton, one of a number of researchers who had implemented and run 5^th Dimensions for some time

1. A 5^th Dimension is a joint project between a university/college and a community institution. The university provides supervised undergraduates to the community institution as labor while the community institution provides necessary space, equipment, and supervision of the activities to provide the students with a valuable research experience..
2. The activity is a mixture of "leading activities" (as proposed by cultural historical activity theorists) including affiliation, play, learning, and work.
3. Participant structures are designed to minimize power differentials between the participants, particularly the undergraduates and the children with whom they work.
4. Heavy emphasis is placed upon the value of communication in a variety of media including computers, conversation, and writing in the service of solving goals that are provided within the activity setting.
5. Participation by children in the activity is voluntary. Children are free to leave at any time. Consequently, the games and other activities that participants engage in must adhere to goals that the children find interesting.

Rising to the concrete

Core principles are clearly necessary for understanding the conditions that promote sustainability and diffusion: after all, you have to have some yardsticks against which to measure deviations from the initial model. However, as abstractions, the core principles are completely empty. They only exist in their embodiments. And it is in their embodiments that we encounter them. So lets take another look at the core principles as they have exhibited themselves in a few of the 50 or so 5^th Dimensions that have come into existence over the past decade and a half.

First, consider the requirement of a partnership between an institution of higher learning and a community organization. The particular quality of activity that emerges as a 5^th Dimension depends crucially on what kind of institution and what kind of community organization enter into partnership. If we restrict ourselves only to the United States, there are now more than a dozen such partnerships in existence. They involve a large number of research universities, two private colleges, two community colleges, and two State Universities. Within these different institutions the departments which send undergraduates to community sites include psychology, education, linguistics, communication, and human development. Variability on the community side is equally great. Community hosts have included youth organizations, schools, libraries, and a church. Each of these combinations yields its own unique context and unique set of possibilities for organizing activity within that setting. For example, if the site is run at a school, provision and maintenance of computers is likely to be less of a problem than if the activity runs at a church.

The combination of partners also impacts the range of activities that children can legitimately engage in and the forms of behavior deemed acceptable. For example, rules of decorum in a school or library afford quiet attentiveness but discourage play while rules of decorum at a Boys and Girls Club may afford a great range of possibilities for engagement but also constant invitations to become distracted from the task at hand. Alternatively a program that runs in a church as the only afterschool activity can invite and depend upon direct parental involvement and support, while a program that runs in a Boys and Girls Club or an afterschool program in a rural area where parents pay to send their children because they are not available to be with them cannot count on this potential source of support. Such factors are crucial to shaping how play, learning, and other leading activities will be combined in each individual case.

Nor do such factors depend entirely upon the institutional identity of the partners. During one period I ran three 5^th Dimensions from a single class at UCSD, all in Boys and Girls Clubs in demographically similar suburban towns in the neighborhood. Yet the local systems that developed in each location differed markedly as a consequences of such features as the amount of space allotted to the activities and the other activities which were available concurrently.

The issue of participation structures and power differentials is also greatly affected by the combination of sponsoring institutions. If university students come from a psychology or linguistics class, for example, their expectations for their role in the activities is often different from those of students who are engaged in teacher training programs. The latter often find it difficult to give up the role of teacher and participate with students, undermining the principle of minimal power differential. In terms of geographical variation which carries with it sociocultural variation, those preparing to become teachers in California come to the activity with markedly different expectations than otherwise equivalent peers in North Carolina.

Several factors provide resources for enacting the principle of leveled power relations. One strategy that has been adopted by many sites is to invoke a mythical creature, a Wizard, Maga, or Golem, as the ultimate arbiter of disputes. By mediating disputes through such a figure, power is removed from the immediate situation and mediated through discussion in the university class and the fantastic/playful character of the magical entity, allowing a collective solution to the problem in place of direct power assertion. The fact that the children often participate for longer periods than the undergraduates also helps to level the playing field since they are often more expert at the game, and know more about the local culture of their 5^th Dimension than the undergraduates, who arrive on a 10-15 week schedule. There is nothing like being instructed in how to find Carmen San Diego by a 6 year old to shift an undergraduate’s attitude in a hurry.

Fulfilling the principle that forefronts the use of a variety of communication means, along with leveling of power relations, is one of the central pedagogical principles governing the 5^th Dimension as well as a practical tool for engaging the children’s voluntary participation in educational activity during afterschool hours. Practically speaking, children are attracted by the computers and other games as well as the presence of undergraduates who devote attention to them. (Theoretically, we invoke Vygotsky’s principle that "the thought is completed in the word" or Piaget’s emphasis on reflective abstraction as an engine of cognitive development to motivate these practices). The ways in which this principle is embodied in the activities varies enormously from site to site, depending upon all the factors mentioned thus far. Most sites make use of specially designed "task cards" which provide intermediate goals and insert the need to stop, reflect, and communicate, into the playing of various games. Other sites make use of their local Wizard-like entity to engage the kids in letter writing and analysis of the games they have been playing. Still others include the requirement to write hints to a community "hints book" to ascend to a higher level of competence in a game. All of these practices both encourage reflective abstraction and communication, and provide a model of how a community builds useful knowledge that the undergraduates as well as the children come to appreciate when they encounter a game they have no idea how to tackle.

The principle that children should not be forced to engage in the activities of the 5^th Dimension is more than a practical recognition that these afterschool activities are not mandated by the state. As Alexander Luria argued more than 70 years ago, it is only when psychologist and subject are engaged in voluntary, coordinated, joint activity that the psychologist has a genuine opportunity to know what the subject is thinking (Luria, 1932). The voluntariness and jointness of the activities is the source of enormous positive affect, a condition akin to Csikzentmihalyi’s notion of flow. It produces conditions which get the undergraduates as well as the children to come back for more, producing affective bonds that can become so strong after as few as 18 meetings over 9 weeks that we routinely spend undergraduates’ last day at site engaging in a ritual "going away party" which often fails to prevent the flow of tears from younger and older participants alike.

When taken in their entirety and embodied in living systems, the implementation of these core principles produces an amazing variety of local variants. There are 5^th Dimensions where bilingualism is the norm, others that are monolingual. Telecommunications plays a heavy role in one place, but not another. Parental involvement is crucial in some places but not others. Some sites make exploration of the local community an important part of their activities, others are more self contained. It is this openness to local variation that appears to be essential to the success of the 5^th Dimension in catching people’s imagination and providing them with a sense of local ownership so necessary to getting the activities going.

I have always envied Ann and Joe’s ability to make relatively standardized tests, with pretest and post test and control group as a part of their deign experiment-oriented research. The question of evaluation when one conducts an afterschool program is much more complex.

First of all, as I indicated, we want these activities to be, in some non trivial sense voluntary. Minimally, this means the freedom to come and go as you please. And minimally, this is a nightmare for standard experimental designs! What if your post test falls on a day when the site coordinator is ill, so the organization you work with decides there will be no 5thdimension that day? It not only means you have failed to get "clean" pre-test/post-test data on the kids, you have failed to provide your undergraduates with the experience you had designed with them, too, in mind.

We have been very fortunate in the variety of methods that people have worked out to demonstrate, in some quantifiable form, that they are doing good for kids. In many places, core games are introduced early and late in the year simply to mark improvements in basic skills. In some places, it is also easy, at least for a while, to get clean tests of the systems’ efficacy. However, in my own institution, for example, record keeping is an endemic problem, and after 15 years of effort, we have come to recognize that we are in a constant process of developing as the environment around us changes.

Now that afterschool has suddenly begun to attract a lot of social attention, there is a pretty strong literature on the efficacy of enrichment-based afterschool educational activity. Except in extreme environments (where, for example, men are hostile and aggressive with tean age males) being in benign afterschool programs is generally good for achieving the traditional role of the schools; kids do better on their test scores.( For evidence concerning this issue see Blanton et al., 1997; Mayer et al. (1999)).. So far as I know, there has been no "one right way" to get the systems properties together. If my understanding of such systems is correct, there never could be a single closed set of prescriptive rules. So, local variability holds a strong hand of cards.

Notice that by virtue of the fact that we started with what we asserted was already a successful educational innovation, focusing instead on sustainability, we were led naturally into discovering something about dissemination. We know that the 5^th Dimension can function successfully in a great range of environments (keeping in mind that some of the systems are stronger than others in producing controlled "proofs" of their efficacy).

But what about sustainability of programs demonstrated to be successful in the places where they have been created, developed, and flourished? It is important to know what happens when the original money disappears. Is death of the system an inevitability? Can we take away some lessons about the conditions that are likely to influence the course of development, in general, and not only in moments of self-conscious crisis?

It is still too early to provide a solid answer to the first question although one of the three 5^th Dimensions that I set up in 1987 in suburban San Diego is still in existence, and by many criteria, still going strong. On the other hand, it is only in the past few months that an originator of a 5^th Dimension has left his University, leaving it to other colleagues to continue, or discontinue, the system. Hence, it is important to note that nowhere has there been institutionalization of the 5^th Dimension in the sense that it continues as a routine practice of its constituent partner institutions in the absence of an initiator.

However, we have learned a good deal about the conditions under which such systems can fail, even when the original money is still present. Nor does disappearance of the initiating funds spell necessary doom for the system.

Here are some of the reasons that appear to emerge as critical events for the life of the system on the basis of our experience:

1. One or more of the partners may discover incompatibilities they did not realize at the outset. Despite honest efforts all the way around, until one has accumulated experience trying to conduct joint activity, one does not know the transaction costs and the difficulties of maintaining the activity. In some cases an institution of higher learning realizes that offering a class every quarter or semester is simply too difficult/expensive to arrange for, even if it appears a wonderful educational opportunity for its students. In other cases, a community organization may discover that they do not approve of children operating on relatively equal power relations with adults.
2. The environment of the activities is dynamic. This dynamism may work either to the advantage or disadvantage of sustainability .

1. Anyone who has worked at a university for any number of years becomes impressed by the very short memory span of the institution. A new department chair may have her own priorities, regardless of the quality of the program. The discontinuities in personnel and levels of funding characteristics of afterschool community organizations are a constant threat to their ability to remember, let alone follow through on, their original intentions.

b) Dynamism can also lead to growth and local sustainability. At a time when the local 5^th Dimensions at UCSD were struggling because of disappearance of start-up funds and competing demands on our teaching time, a crisis at the State level reorganized and expanded our local activities. This crisis was caused by the abrupt termination of affirmative action programs and the need to demonstrate the University’s commitment to outreach. This crisis in the larger context also provided the opportunity for expansion of the system (all branches of UC now run 5^th Dimension programs) and for local sustainability (the statewide program, called UClinks, obtained permanent state funding at a level which allowed its continued functioning).

But this stage of the research is by no means run its course. Just because a system is sustained in name does not mean that it is sustained in life. Hence, methods of evaluation must be found which allow us to quantify the degree to which the key properties of the basic model are present and to what degree. But we have to be able to do this, as I hope my discussion has convinced you, in constant tension with the principled heterogeneity and local autonomy of local systems. Ann was right in pointing to the expanded level of social intelligence that would be needed to address these issues. We are dealing here with an inherently multidisciplinary object, to be studied psychologically, anthropologically, and sociologically at the same time. Coordinating the insights of these different approaches seems to be the central challenge in design experimentation for the study of learning and development. It is not an issue I will solve in this talk, but it is one I hope to live long enough to address.

References

(cited or implied)

Belle, D. (1999). The after-school lives of children : alone and with others while parents work. Mahwah, N.J. ; London : Lawrence Erlbaum Associates,

Blanton, W. E.; Moorman, G. B.; Hayes, B. A.; Warner, M.L. (1997).. Effects of participation in the fifth dimension on far transfer. Journal of Educational Computing Research, 16 (4):371-396

Brown, A. (1992). Design experiments: Theoretical, and methodological challenges in creating complex interventions in classroom settings. Journal of the Learning Sciences, 2,141-168.

Cole, M. (1996). Cultural psychology: A once and future discipline. Cambridge, MA: Harvard University Press.

Cooper, H., V, Jeffrey, C.; Nye, B., Lindsay, J. J. (1999).Relationships between five after-school activities and academic achievement. Journal of Educational Psychology, 91 (2):369-37

LCHC (1982). A model system for the study of learning difficulties. Quarterly Newsletter of the Laboratory of Comparative Human Cognition, 4(3), 39-66.

Mayer, R. E.; Quilici, J. L.; Moreno, R. (1999). What is learned in an after-school computer club? Journal of Educational Computing Research, 20 (3):223-235

Pierce, K. M.; Hamm, J. V.; Vandell, D. L. (1999). Experiences in after-school programs and children's adjustment in first-grade classrooms. Child Development, 70 (3):756-767

Posner, J. K.; Vandell, D. L.. (1999). After-school activities and the development of low-income urban children: A longitudinal study. Developmental Psychology, 35 (3), p.868-879