Ideas Going Public

Yesterday I submitted the following article to 'Research in Higher Education', an above first-tier journal in Memphis, TN. Let's see what they say.

Rethinking Accountability and Quality: The View from Complexity Science
First Author:

Sean Park
Instructor and Research Associate
Bachelor of Health Sciences Programme, Faculty of Health Sciences
McMaster University
Hamilton, ON
Send correspondence to: sean.park@learnlink.mcmaster.ca


Second Author:

Del Harnish
Assistant Dean, Bachelor of Health Sciences Programme, Faculty of Health Sciences
Professor, Department of Pathology and Molecular Medicine, Faculty of Health Sciences
McMaster University
Hamilton, ON
harnishd@mcmaster.ca


Submission Copy

Rethinking Accountability and Quality: The View from Complexity Science
Abstract

Accountability and Quality, if they are to be useful guiding forces for improving education, must account for the complexity of social interactions. It is the view of the authors that human behaviour is influenced by local conditions, and as a result of past histories and many interactions over time, this behaviour is non-linear. If we are to understand outcomes in the context of Accountability and Quality, viewing ourselves as local agents in Complex Adaptive Systems (CAS) is useful for understanding how local conditions, interactions and history influence the human interpretation of instructions and evidence. This paper offers an accessible overview of CAS and discusses the implications of complexity for measuring and interpreting educational outcomes.

Keywords
complex adaptive systems; complexity; accountability; organizational effectiveness; organizational culture; leadership; education administration;

There is a growing movement in the social sciences that characterizes human behaviour and the social systems we create as Complex Adaptive Systems (CAS) (Buckley, 1998; Byrne, 1998; Dooley, 1997; Gell-Mann, 1994; Holland, 1995; Kauffman, 1993; 1995; Lewin, 1992; Marion, 2002; Zimmerman, Lindberg, & Plsek, 1999). Social organizations and systems, such as cities, communities, social networks, and economies are complex interwoven webs of dynamic relationships with patterns that set the context and framework for how we make meaning of our interactions with each other. The struggle to understand these interactions and how they impact the functioning of the greater whole, however we might define that whole, is a common one. We have come to discover through our experience that one way of thinking about these interactions has been through our study of CAS. CAS as a framework for describing human behaviour, we argue here, is useful because CAS allows us to place the role of our personal experiences, perspectives and the particular cultures in which we work at the centre of understanding changes in human behaviour.

‘Accountability’ and ‘Quality’ are concepts that have come to characterize the discourse of improving education in North America. On the surface it appears that these concepts speak to the need for mechanisms that ensure resources are used wisely and that students are getting the best possible education as supported by evidence about effective educational practices. Many of us are familiar these mechanisms and their impact on schools, colleges and universities. At the post secondary level, a number of Canadian universities have recently signed on with the National Survey on Student Engagement (NSSE) in an effort to be more accountable to providing a high quality undergraduate education. NSSE measures an institution’s performance along dimensions of student perception framed by Chickering and Gamson’s (1987) Seven Principles for Best Practice in Undergraduate Education. At the provincial level in Ontario, the Education Quality and Accountability Office (EQAO) tests students at various stages of their schooling to determine who is performing above or below grade level and how institutions are performing relative to each other. Schools in the U.S., in at least one respect, have had to contend with the No Child Left Behind Act (NCLB) whereby funding is tied to continuous improvement in school performance.

The goals are sound, yet the ways in which we are using these accountability mechanisms can often be counter-intuitive and counter-productive. In an effort to ‘motivate’ better performance there are many unintended consequences. How do students feel about themselves after being told they are not performing at ‘grade level’? What incentive exists to motivate change if an institution comes out on top of an evaluation? How does punishing poor performance foster a willingness to try new things? The problem with accountability and quality, alluded to by Finnie and Usher (2005), is that “there is no ‘silver bullet’ in looking at educational quality. There are no simple measures you can point to and say ‘Yes, there is quality. Let’s have some more of it.’” (p.9) What it means to be accountable or deliver a high quality education is so utterly dependent on how one defines those concepts in context of the specific issues one faces that it is perhaps better to shift our thinking to understanding how people respond in localized ways to the broader efforts aimed at making education more accountable or of better quality. Taking this approach to rethinking accountability and quality is outlined in context of the uncertainties that come with trying to predict how successful our practices as administrators and educators will be and the desire to find a ‘science’ to administration and teaching. Finding a set of evidence-based best practices to implement in our institutions and holding those institutions accountable to those practices, it seems, would serve us well. Our perspective is that such an approach to improving teaching and administration in higher education, while understandable as a reaction to the uncertainties of practice outcomes, does not help us answer why, for example, evidence-based strategies to improve teaching and learning employed by institutions with the same resources in the same community produce drastically different results, or no results. Or, despite the apparent evidence on how to improve teaching, why our best efforts to control for all the variables, including uncertainty itself, so often fail us in making systemic changes to improve education. We have come to understand that questions of best practice and measuring outcomes for accountability purposes are more appropriately asked from a framework that has parity with the view of human organizations and interactions as complex and non-linear phenomena. The science behind CAS, referred to as Complexity Science, may provide such a framework and our first goal of this manuscript to offer a digestible primer on CAS. We do not aim to be exhaustive with the primer, and as such, do not address the current scholarly debates taking place around the ‘defining’ properties and conditions of CAS. With a CAS framework established, our second goal is to address implications for researching the behaviour of individuals and groups in educational settings as CAS.

From Chaos to Complex Adaptive Systems

To begin our discussion of administration, teaching, and learning as CAS, we must put Complexity Science in context of its cousin Chaos. Many of us are familiar with Chaos theory vis-à-vis Lorenz's (1993) 'butterfly effect'; small, immeasurable variations in the initial conditions of complex systems will be re-iterated over time creating non-linear processes and unpredictability about the precise state of the outcome. In other words cause and effect are not proportional because small, unnoticeable differences in the conditions of the system can produce very different outcomes. This effect might suggest that the behaviour of systems with many interacting variables cannot be controlled or predicted, but the presence of a 'strange attractor' shows us that there is order within the 'Chaos'. Strange attractors refer to points or patterns of stability within or around which a system is likely to dwell. Wheatley (1992) suggested that values such as quality or behaviours that embody those values might be thought of the attractors in human organizations. In our discussions with educators, however, the aforementioned description is about as much as many of us have learned about Chaos theory or its application to social systems. This may be unfortunate, but somewhat understandable from the social science point of view. Chaos theories come primarily from research in the physical sciences, mathematics and computer science (Gleick, 1987; Lorenz, 1993; Mandelbrot, 1983; Prigogine & Stengers, 1984; Wolfram, 2002). These scholars have unearthed the assumptions we have about the world when viewed from a traditional ‘Newtonian’ perspective and paved the way for introducing the concepts of non-linearity, order from chaos, Lorenz’s ‘butterfly effect’ as manifestation of sensitivity to initial conditions, fractals and emergent properties. These concepts have been subsequently been re-interpreted for social systems, although there have been significant cautions against it’s misapplication. Prigogine and Stengers (1984) cautioned against the way in which social and economic theorists have traditionally applied the concepts and methods from restricted domains of study in physics to social phenomena. The social sciences have often imported the Newtonian approach in physics where certain kinds of phenomena, like motion, are quite predictable and linear. Predictability, if it could be harnessed in the social world, would be the key to control. Social phenomena, however, is frequently unpredictable and certainly non-linear. Therefore it is theoretically inappropriate to use models that are inconsistent with the complexity and non-linearity of human interaction. The argument has been advanced that we should differentiate between physical systems and social systems by recognizing the difference between the concepts of Chaos and Complexity. Marion (1999) remarked that:

Complexity theorists, who are conceptually related to Chaos theorists, argue that life tunes Chaos’s intensity down a bit to a transition band between Chaos and predictable stability called the Edge of Chaos. Dynamics in this band are still Chaotic but they also possess characteristics of order. Full-blown Chaotic systems have little memory; living systems must be able to map their past. Chaotic systems fit a bit too readily from novelty; living systems need to consolidate gains. Predictable, stable systems, by contrast, possess none of the panache needed to created new order or even to respond adaptively to creative environments. Complex systems lie between these poles, at the Edge of Chaos, and they have both panache and stability sufficient to serve life. (p. xiv)

Theories of Complexity then, must be distinguished from those of Chaos, and in seeing the difference, we can begin to appreciate that some types of systems have both elements of unpredictable and predictable behaviour. Both are required for purposes of internal coherence and adaptability to changes from within or imposed by the external environment. Systems that dwell in and around this so-called 'Edge of Chaos' have come to characterize where CASs are most adaptive (Marion, 2002; Zimmerman, Lindberg, & Plsek, 1999). And herein lies the key to making sense of uncertainty. Understanding how complex systems are wont to behave is about identifying the necessary and unavoidable processes that produce unpredictable behaviour just as much as the variables that keep the system coherent and stable. CAS theory is valuable in helping us determine what systemic processes establish coherence and why these are the same processes that produce the uncertainty necessary for system stability. Let us see if we can show what CAS looks like when applied to an example of complex social behaviour that is not described in terms of CAS. Consider a snapshot of the work of honeybee expert Tom Seeley (1995):
When a hive of honeybees grows beyond its ability to be contained within its current nest site, the hive splits off and a swarm forms on a nearby tree branch. Unprotected from elements and without a nest in which to make honey to sustain themselves, the bees will perish within a few days. The decision-making process for where to set up a permanent new nest is dependant on a strong set of diverse options because options are weighed relative to each other. Without a good range of options to choose from, the bees will potentially make a mistake and choose poor sites over good ones. Scouts are assigned to survey the environment and make yardstick measurements about the sites they come across and return to the swarm indicating to bees in their local proximity, the distance from, access to, and protection afforded by a potential nest site. Swarm bees may decide to either accept or reject a site communicated by a scout, but once they accept, they commit to the decision and begin ‘dancing’ at a higher frequency.

Relatively desirable options gain momentum in the swarm, and once enough critical support for a site has been reached (when enough bees are ‘dancing’ to push the swarm past a critical temperature threshold), the swarm moves en masse. Although each bee has the autonomy and capacity to make a decision about whether or not they are willing to commit to an option, there is no one bee tallying up all the individual decisions. It can be thus be said that the selection of a particular nest site is a function of ‘collective intelligence’, emergent from the process of local communications between scouts and swarm bees. Once a sub-critical to supra-critical temperature shift has taken place, the global dance signals to each local bee that take-off is imminent.

The swarm of bees is a CAS - complex because there are a multitude of known and unknown variables that are interacting to produce the behaviour; adaptive in the sense that the system has the potential to change and learn about what the next 'best' step is; and a system because it is composed of interacting agents (honeybees) guided by different kinds of rules. The case of the honeybees is quite suggestive of what we do as humans in solving problems. As educators we have some good ideas about what is of value (i.e. sustained contact with students). Even in a hive, what constitutes a good site will vary between scout bees just as we have our own differences in perception. The rightness or wrongness of a value does give us instructions of how to proceed and, in the end, how humans come to understand and construct knowledge is about how we experience the world as both a subjective and inter-subjective human interacting in a context with other humans. The terrain beyond the swarm is full of possibilities and given the time of year, the number of bees, and a multitude of other conditions, how assessments gain or lose momentum will depend on these contextual factors. The question of implementing best practices now becomes one of how individuals and ideas are relating to each other in context. How this relating occurs to create meaning and suggest choice necessitates a deeper look at CAS. The literature on CAS is rich with multiple, yet related, theories about the finer details of CASs and how they operate. (Brockman, 1995; Capra, 1996; Casti, 1994; Cilliers, 1998; Dooley, 1997; Goertzel, 1993; Goodwin, 1994; Holland, 1998; 1995; Kauffman, 1995; Stacey, 1996; Waldrop, 1992). The following is a synthesis of the key features of CAS drawn from this literature and, where appropriate, is related back to the honeybee example. In the language of CAS, the parts of a system are referred to as agents. The agent will vary depending on the system being studied. It could be a human being within a community, process and programs in an organization, geese flying in a V-formation, the honeybees in our previous example or a neuron within the fabric of consciousness. To be classified as an agent in a CAS, it must be interdependent with other agents in the system and be able to make independent choices and actions based on changes in the moment-to-moment conditions it experiences. In this way, the two key qualities shared by all agents within any given CAS are interdependence and autonomy – one of the many paradoxes in CASs. Additionally, the actions of agents are thought about in terms of local conditions and referential knowledge. By this it is meant that agents act in the context of their past experiences, the immediate circumstance, and the local rules, norms and information from their environment. The action of individual agents, however, does not help us to describe the overall behaviour of the swarm, which is directed at finding a high-quality nest site as quickly as possible. The decision is not made by one honeybee and is a collective decision. The ‘social intelligence’ displayed by the swarm is a property of the system that is not ‘encoded’ within the rules governing the behaviour of the bees – it comes about as a consequence of their complex interactions. In the language of CAS, this is referred to as emergence. There is often an implication that emergences are unexpected or unpredictable. The nest chosen by the bees is a good example of an emergent decision developing out of a highly organized, complex and decentralized set of interactions with many autonomous and interdependent participants using a common language. An important point about predictability must be made here. Although the particular nest site chosen is not a predictable outcome, the nest selection behaviour is predictable because it is genetically coded. CASs are not governed by ‘master’ agents or universal rules that control the system in a centralized, hub and spoke, or top-down hierarchical model (though these controls can still affect CAS behaviour). Instead, roles within a CAS are emergent properties of the system such that internal organization and leadership emerges out of the relationships between agents and their effects on each other.

CASs can form and evolve when there are basic rules for a system, often referred to as minimum specifications. ‘Min Specs’ are thought of as the minimal amount of information or instructions required for the system to maintain viability. We would expect all honeybee swarms to use the same sets of basic rules to survey the environment and relay information back to the swarm to make a collective decision. These rules do not dictate what the final outcome will look like as it is the self-assembly or self-organization of bees through following and interpretation of these rules in context of local conditions and referential knowledge that will determine how and what decision gets made. A swarm of honeybees, of course, is not an isolated system. CASs are always embedded within other CASs and we can see that embedded within the bees are various social, biological, chemical and physical systems. The bees are also embedded in the broader ecosystem, which includes a variety of biotic and abiotic factors. Systems that exist within a series of nested systems can have self-similarity across levels. Self-similarity means that patterns at one level of magnification are similar to patterns at other levels. A result of such nesting means that CAS subsystems and suprasystems evolve together and in response to one another such that no part exists in a vacuum and any changes have results seen throughout the system. In order for CASs to adapt and maintain organization, there must be a critical level of diversity. Diversity is a function of both the variance and number of agents, parts, or options in a system. We can talk about the diversity in the honeybee swarm in terms of the different nest site options and the different scouts surveying the environment. Generally speaking, systems with low diversity are less more vulnerable to collapse than systems with high diversity. There are limits, however, to the optimal amount of diversity in a system. If so many honeybee scouts were bringing information from the environment that the swarm was overwhelmed with information, it would be reasonable to assume that the collective decision-making capacities of the swarm would be compromised. Understanding the role of diversity in a CAS should be thought of in tandem with the interactions and relationships taking place between agents. Interactions and relationships can be described in a number of ways. We might speak of the communication or contact between two or more agents in which a gesture invokes a response. The scout bee needs to interact with bees in the swarm to share information from the environment. Depending on how the interactions are allowed to occur in any given CAS, the flow of information through the system will be greatly affected. This is important because if either the agents are not diverse enough or there are not enough interactions taking place between the agents, then the system will not have the ability to successfully adapt to changes in the environment.

Adaptability is shown in the honeybee system as a function of the diversity and relationships between the bees rather than something conferred on the system by the executive powers of privileged agents. Adaptability does not carry a value judgement as it refers to any modifications that assist the organism, species or system to continue existing under new conditions. Modifications may even be made to the rules that govern the system, and in effect, change the relationships between the agents. Because of the complex and diverse interactions in CASs, they are non-linear in the sense that cause and effect are not proportional. We can think of just how complex a system is by recognizing that linearity refers to the degree to which the magnitude of a cause can predictably produce a proportionate effect. In a linear system, a small cause will produce a small effect and a large input will result in a large output. CASs, however, are non-linear because small changes in the system may produce disproportionately large, unexpected outcomes while large inputs do not always produce large outputs. A small rumour running through a school, for example, may cause more change to the system than a school-wide lecture in the auditorium because everyone may return to their classes not having taken anything from the lecture. Non-linearity suggests that CASs are unpredictable, but when we understand how local interactions between agents impacts how those agents interpret and act, we can see patterns and explain why certain behaviours do and do not occur. The rest of our discussion on CAS looks at some concepts that explain why.

Related to non-linearity are two other properties: sensitivity to initial conditions and historical dependence. Sensitivity to initial conditions means that the direction and outcome of a system is affected greatly by the conditions at the start of the system. The initial conditions for the honeybee swarm would include, for example, the weather conditions, swarm size, the energy of the scout bees, and location of the swarm in an ecosystem. The specificity of these conditions impact the trajectory of the system in the sense that the same honeybee swarm on a different day or a different part of the woods will have a slightly or drastically different decision making process. In most respects, we do not have the tools to accurately and precisely capture the initial conditions of a system. We are forced into making approximations, like rounding numbers up or down or describing what we see in terms of familiar categorical frameworks. However, because the initial conditions help set the direction of future trajectories, thinking about how those conditions shaped the system is helpful for understanding the systems behaviour.

The behaviour of any complex system is influenced by the past. Historical dependence means that interactions and patterns in CASs are iterative and learning occurs as a result of feedback and experience. The honeybee is tied to it’s own history through evolution in which the genetic map, when looking back in time and in comparison to other species, helps one to see a snapshot of past interactions that have come to shape what the honeybee is and does. At a psychological level, feedback and experience come to bear upon how we as humans learn and respond to our environment. In his work on how adaptation yields complexity, Holland (1995) described how CASs, like human minds, use combinations of ‘building blocks’ that have been tested with each other over time. Some combinations of building blocks will confer an advantage over others, so it is advantageous to the CAS to ‘remember’ what combinations work.
Wherever we turn, building blocks serve to impose regularity on a complex world. We need only look at the use of musical notation to transmit the endless variety of music…The point applies with at least as much to our everyday encounters. If I encounter “a flat tire while driving a red Saab on the expressway,” I immediately come up with a set of plausible actions even though I have never encountered this situation before. I cannot have a prepared list of rules for all situations…So I decompose the situation, evoking rules that deal with “expressways,” “cars,” “tires,” and so on, from my repertoire of everyday building blocks. By now each of these building blocks has been practiced and refined in dozens or hundreds of situations. When a new situation is encountered, I combine relevant, tested building blocks to model the situation in a way that suggests appropriate actions and consequences. (p. 37)

Perhaps one of the most visible features of CASs is that they all exhibit decentralized control, even if control appears to be applied centrally. Consider the decision-making process that evolved in Warrior Ants (ecitron burchelli). Warrior ants are nest-building insects that that swarm behind leader/scout ants and occupy any potential nest site they come across. These behaviours lead to poor nest site choices, the cost of which is demonstrated insofar as Warrior Ants have to constantly find new nest sites, making themselves vulnerable to predation and the elements. Warrior ants live in the Amazon rain forest where there is no shortage of nest sites, but all are of relatively low quality. Such systems are wasteful and perilous yet highly effective in certain settings; after all, throughout millions of years, Warrior Ants are still with us and show no sign of becoming extinct. An important observation needs to be made here. The absence of any evident adaptability that may arise from not requiring ‘participatory’ or decentralized control to survive does not mean these systems are not complex, adaptive, non-linear, nor having decentralized control. Anderson and Franks (2001) believed that it is “deeply misleading to refer to the ant at the front of the team as a leader in any sense that implies the special role of organizational leadership. Nor is such an individual an ‘organizer’ or a ‘key individual.’” (p. 537). They further stated that the roles of leader and follower in ant colonies appears to be the consequence of how teams form to achieve certain tasks and probably help to establish a division of labour in the team. Finally, when applied in the context of human organizations, CASs bear legitimate and shadow subsystems contained within any set of rules governing the system and are often referred to in the context of the Gestalt model of organization (Egan, 1994; Stacey, 2001). The legitimate subsystem is the status quo, the official rules and behaviours enacted to serve the primary task of the system. Guidelines, policies, and mission statements are good examples of legitimate subsystems in organizations. Shadow subsystems are the informal organizational norms that capture how people are actually communicating, interacting and giving or following instructions. Shadow subsystems may work to subvert official protocol or modify the dominant subsystem to more accurately reflect the realities of local interactions. In summary then, the key parts of Complex Adaptive Systems are:

1. Agents and parts have autonomy and interdependency and act based upon local conditions and referential knowledge
2. Agents can self-assemble with minimum specifications
3. CASs are embedded within other CASs
4. Control is decentralized throughout the system
5. Organization and complexity emerges from the relationships between the agents and parts
6. Exhibit self-similarity across levels
7. Exhibit non-linear behaviour
8. Sensitive to initial conditions and diversity in the system
9. Trajectories and patterns are affected by historical dependence, feedback and experience
10. Have legitimate and shadow subsystems

Education as Complex Adaptive Systems

CAS is a powerful framework for understanding what we might consider ‘living’ systems, but why should we use it to think about human interactions in educational organizations? The short answer to this question is that, as humans, we are living organisms and if we are to describe ourselves and think about how we interact, it follows that those descriptions and theories should be congruent with our best understandings of how humans think, learn and interact. From neuroscience and psychology, to social network theory, and into the natural ecologies in which humans are interdependent, complexity-based perspectives are emerging in these fields because they elucidate many of the dynamics, interactions and behaviours of complex phenomena. To garner a fuller understanding of why education is more appropriately understood from a CAS perspective, we can look to the application of complexity in the social sciences and organizational theory (Axelrod & Cohen, 1999; Brown & Eisenhardt, 1998; Buckley, 1998; Byrne, 1998; Marion, 1999; McKelvey, 1999; Stacey, 1996; Weick, 1996; 1997;); business (Clippinger, 1999; Pascale, Millemann & Gioja, 2002); health care (Anderson, Crabtree, Steele & McDaniel, 2005; Zimmerman, Lindberg & Plsek, 1998) and education (Cutright, 2001; Davis & Simmt, 2003; Davis & Sumara, 1997; Sanders & McCabe, 2003). In thinking about education as a CAS, we have learned to appreciate accountability, quality, and changes in practice in terms of how simple, complicated and complex the particular phenomena in question is. Zimmerman (2002) differentiated between simple, complicated and complex with the analogies of following a recipe, sending a rocket to the moon and raising a child.

Following a recipe is simple. It may not always be easy as some recipes have many steps and procedures but it is a matter of following instructions carefully and mastering techniques through practice. Sending a rocket to the moon is complicated. Although some aspects of the procedure are simple, there is more to it than following a whole series of “recipes”. There is the additional challenge of coordinating experts and expertise from a variety of disciplines. The coordination skill is not as simple as following a recipe. (p.9)

Raising a child, however, is complex because there is a co-evolutionary relationship between parent and child. Zimmerman (2002) reminded us that “Raising one child is no guarantee of success with the next. Each context is unique and there is no rule book that works in an absolute sense." (p. 9). Despite the uncertainties, she also pointed out that there are patterns in child rearing we see across generations, geographies, and cultures. Differentiating between simple, complicated and complex systems helps us to think about what sorts of outcomes we can expect to see when we try to measure individual and institutional behaviours.

If we are trying to measure the quality of a new building, for example, we might look at how well the contractors followed safety codes and design specifications. This is a rather simple assessment with a high degree of certainty about what is being measured and what the outcome means. A campus-wide audit of building quality is much more involved and would be considered complicated because of the co-ordinating skills required. We can still have clear expectations about how the outcomes are going to be defined and what they will likely look like. However, because social interactions are characterized by non-linearity, sensitivity to initial conditions, and local agent interpretations, future outcomes describing the quality of socially situated activities will necessarily be ‘fuzzy’. Fuzzy infers that what we see ahead of us has shape, but is ill defined because our still distant perspective does not allow us to differentiate between the details. As we approach, the details begin to fill in and when we arrive, perhaps we are fortunate enough to be able to look in the rear-view mirror and catch a glimpse of how and what kinds of social interactions help explain the outcomes before us. In the world of the honeybees, the nest site that is chosen by the swarm cannot be predetermined.

In approaching simple and complicated systems as linear sets of relationships, we have a handle on predicting future specificity. Stacey (2001) argued that we have to move away from trying to conceptualize human systems in an objective, linear way because, as CAS suggests and experience teaches us, human interaction is frequently non-linear. Additionally, in being 'agents' in the same systems we are trying to study, we are both constrained and enabled by our own perceptions and mental models depending on where we stand. 'Systems thinking' as it is called, leads us to believe that we can look down from above at the world, like we can in CAS computer simulations. This is an ambitious, if not all together impossible approach if we are the agents engaged in the very processes we want to study. We do not have omnipotent ‘agency horizons’ as Homer-Dixon (2001) called them; our ‘reach’ in time and space has limits. The key is to let go of the idea that we can conceptualize and work with whole systems and “focus attention on local interactions between people” (p. 7). How honeybee swarms select a nest site does not happen because the bees might somehow have a global perspective of the swarm, but because of the knowledge that is created in the local interactions between bees. Using CAS theory to understand education means seeing ourselves as locally inter-acting human beings. Like the patterns we see across cultures with raising children we can have some certainty about knowing whether or not educators and administrators are providing the highest quality of education possible. Frequent contact, a sense of safety in the learning process, and proper resources are universals for personal growth. In raising a child, however, there is no certainty that following a set of guidelines will ensure success. Similarly, we should expect that any efforts aimed at accountable education should leave room for fuzzy outcomes. Once the future is arrived at and we can look at the specifics and work to see the trajectories by which certain outcomes came about. It is here that we will see patterns.

- insert table here-
Table 1. Simple, Complicated and Complex

Accountability and Quality as CAS If we are beginning to appreciate the role of complexity and uncertainty in social interaction, how do we begin measuring and studying accountability and quality in terms of CAS? Despite the appeal of CAS theory to offer theoretical and descriptive accounts of complex social phenomena, we are left questioning what kinds of questions, tools, and conclusions we should be asking. In what ways is the system, whether it is a classroom, school or district, adaptable? How do we measure adaptability? What roles do diversity and interaction between agents play in the life of the organization? How do we identify non-linearities between cause and effect in organizations? How do we identify and describe the emergent properties of educational interactions? If people act according to their local conditions and referential knowledge, what role do leaders play in influencing change? If we know some of the initial conditions and history of an organization of people, how might we expect people to react under directives to change? Many of the tools we are accustomed to using for observation and measurement draw too heavily on assumptions about the nature of human interaction. We are not simple or complicated beings and questions that neglect to account for our own complexities will skew how find and interpret answers. Why so? Because laws governing behaviour are not universal and outcomes are highly context-dependent, capturing snapshots of a system should provide some account of the context and interpretation of local conditions and rules by agents. The context of human interaction is laden with emotion, localized and diverse perspectives, and capturing how these contextual factors affect agent behaviour requires ways of identifying the specific elements of CAS (i.e. nonlinearity, historical dependence). To do so we need many tools, qualitative and quantitative alike, to capture local context, interpretation, dynamics and patterns (see Smart (2003) for one approach to measuring cultural and leadership complexity in higher education).

Let us, for example, think of a tabulated data set that shows the teaching evaluation scores given to professors by undergraduate students in a particular faculty over a five-year period. We should expect some variance between years and between professors, and in asking how ‘well’ a faculty performed as educators we might look at our data set and see something resembling a normal distribution. A few professors may have performed abysmally to mediocre a few, if not many, come out exceptional. Typically, if we want to get an overall sense of teaching quality, those that scored multiple standard deviations from the norm would likely be excluded from the analysis so as to not skew the mean and yield an ‘average’ performance figure.

People, however, are not anomalies and their performance is both a function of their individual efforts and the socio-cultural context in which they are performing. By removing outliers from the data in this way, we lose the potential for understanding how the system is working or changing. Why are those outliers present? Is the professor who scored dismally low on the evaluations doing something that the students do not expect from a professor, even though the teaching may be of superb quality? Are the professors on the high end doing well because of an already commonly held perception of the professor as a ‘star’? Our point is that because universities, and human organizations in general, are complex and nonlinear, the link between cause and effect cannot be assumed to be a proportional relationship. Normalizing behaviours and outcomes does not help us understand the how the dynamics and diversity in the system contribute to its stability and adaptability.

To get a sense of how approach accountability and quality from a complex systems perspective, we build on Anderson et al. (2005), a work that recognized the need for complexity-based research methods in their attempts to understand nursing homes as CASs. This recognition stems from their observation that despite the plethora of evidence on best practices, substantial improvements in health care are generally not taking place. To better understand the disjuncture between knowledge and action, they advance a complexity-based research framework for studying organizations as CASs. We use Anderson’s work for three reasons. First, we can learn a lot by looking at what is happening in other fields and link them into our discussions on education. Doing so enhances the diversity of our perspectives like honeybee scouts bringing perspectives in from different types of ‘intellectual nest sites’. Second, health care in North America has shifted towards an interdisciplinary approach to health in recent decades because of a growing awareness that the health of individuals and populations is shaped by a confluence of cultural, geographical, historical, physical and psychological factors. Education is much the same in that multiple disciplinary backgrounds come to inform how schools are run, curriculum developed, students supported and how to best teach and learn. Learning and knowledge, analogous to life and health, are the processes and emergent products of interactions between individuals, groups and environments. Knowledge and health, however, can be though of as scaled concepts with ‘poor’ at one end and ‘robust’ at the other.

Learning and life happen regardless of the kind of interactions that happen. If the interactions are right, then the learning will produce good knowledge in the same kinds of ways that eating good food, having social supports, and staying active produce good health. Finally, Anderson’s approach to researching CAS is consistent with the literature on CAS. It offers a reasonable and defensible set of ‘minimum specifications’ for researching CAS environments and interpretation observations in terms of CAS.

Research as CAS

Anderson advances a series of inter-related propositions about how one might go about studying health care organizations using CAS theory to make “useful maps of the system” (p. 169). These propositions are made to address questions such as:
Why have we seen so little change in what is being done for clients despite substantial knowledge in the form of best practice guidelines?, Why is it that a physician who is enthusiastic about preventive services is unsuccessful in delivering them to her patients?, and Why is it that common conditions such as “pain, pressure sores, malnutrition, and urinary incontinence” continue at high rates among nursing home residents. (p.169)

The following is a reinterpretation of the propositions and an attempt to situate our thinking in how to go about to make use of complexity for understanding accountability and quality in education. Before we set out the proposals, we would like to give them an educational context to speak to throughout the rest of the article; the honeybee analogy can lose its buzz after awhile. The account is real and comes from our experience at McMaster University in Hamilton, ON.

Faculty members, in the faculties of Humanities, Social Science, and Science, were given simple instructions as an administrative imperative to complete Final Grade reports for Bachelor of Health Science students enrolled in courses with those faculty members. Health Science students graduate two weeks before other faculties, so grades must be sent earlier than normal to the registrar’s office for convocation. The instructions stated: The final grades are to be handed to the Bachelor of Health Sciences Program, MDCL 3308, by the due date (indicated on the sheet), and a signature is NOT required by the Faculty. Out of the fifty-five faculty members given the instructions, only 29% complied, and only 18% by the deadline stated. Proposition 1: Switch what is foreground and background

Anderson contended that the health care practice environment is traditionally seen as the background, with the physician, nurse or other health care worker as the foreground. If understanding our environments as CASs involves looking at local interactions and patterns of relationships between people, what we see as foreground and background should be reversed. Given this, “the physician’s level of knowledge about something might not be the best place to begin when trying to understand improvements in health care” (p. 670). Similarly, the knowledge of students, teachers and administrators might not be the best places to begin in trying to understand the processes of teaching and learning.

Does this suggest that we are devaluing the role of people in those interactions? Stacey (2001) argues the opposite and remarks that by valuing the context (the organization, the culture, the relationships) of people, we are actually restoring their importance because we are validating the details of their experience and their perspectives. Consider the faculty grade report account above. Such an outcome, of course, would not be likely or acceptable if instructors asked students to hand in a course assignment worth a significant proportion of their final grades. Students are motivated to comply with instructions and deadlines in large part by the penalty exacted if the neither is met to an agreed standard. Although timely grade reporting is important, no penalty was present to force compliance from faculty. For those that did not comply, we had to consider that many of them were accustomed to submitting grades sometime after their courses had ended. Our request likely caught some of them off guard.

Thinking through this experience, we come to see that understanding the context of the people interpreting the instructions is helpful for knowing how the outcomes played out and what they mean. The 'poor' response rate from faculty, in and of itself was not helpful for our cause, but in the larger picture actually turns out to be a very minor issue for us. There are usually only a small handful of students in the program who have exams outside our faculty so late in the term that they need an early grade submission for convocation. We now spend time working on these special cases and less on trying to get compliance from everyone at one time.

Proposition 2: Look for divergence and convergence between ideas and actions

Capra (1996) and Lee (1997) advanced that to better understand interdependencies, we need to look for where ideas and actions succeed or fail at intersecting. For Anderson, this suggested that it is necessary to describe actions in the context of agents’ mental models, which are informed by local conditions and referential knowledge. Both properties help to establish the meanings agents attach to their actions and could help explain why the ‘idea’ to have grades submitted earlier than usual did not translate uniformly into desired ‘action’. By comparing and contrasting what requests are communicated with what actually gets done, one may be able to see, for example, the role communication networks have in enabling ideas and actions to successfully intersect. Mapping the mental models held by agents and their subsequent actions usually requires “prolonged engagement with the system” (p. 673). Prolonged engagement, according to Anderson, can be fruitful because it will help illustrate the discrepancies and consistencies between ideas and actions that are a function of interactions that occur outside the “boundaries of systems” (p. 674). We should expect this, given that CASs are nested within other CASs. Action/Idea analysis is useful for identifying the quality, quantity, and looseness or tightness of the interactions between agents. Faculty members who are clustered closer together and frequently interact to talk about their courses, for example, may account for better convergence between ideas and actions. Conversely, the faculty that did not respond may have never received the message because it was not received or misplaced by a key information agent such as an administrative secretary or a downed server.

Proposition 3: Look for unexpected events and how people respond to uncertainty
The hallmark of CASs rests in their non-linearity. This means that cause and effect linkages may not be good at describing the behaviour of the system because small events can perturb into large ones and vice-versa. Some of the time, or perhaps even most of the time, people may be expected to behave in certain ways, but during periods of instability for example, general models and trends aren’t very useful. If we seek to understand how an organization responds to change, an indicator of it’s ‘fitness’, we should look to unexpected events to see how people react and relationships change in the face of uncertainty.

Looking for where large efforts to enact change result in minimal shifts in the system may be insightful. The faculty/grade example is representative of many initiatives that are clearly communicated, legitimate, and ostensibly reasonable requests for people to alter existing practices for special cases, but end up with mixed results at best. Hiring new administrators or creating new policies may be large change efforts designed to revamp the system, but if teachers intend to wait them out because they know they will be gone again in a short period of time, the existing practices remain unchanged. Perhaps we could have had someone personally go around to all the faculty members telling them about our request, or worked harder to get the request right by putting in more instructions and some incentive. These efforts, however, would likely turn out to be sizable attempts that would yield only slightly better outcomes. Perhaps there is something ‘small’ we could do to produce ‘large’ compliance, but the time and effort figuring out how all the ways in which faculty will respond next time is too great for something that is best dealt with on an ad-hoc basis.

Anderson suggested looking for unexpected events; outcomes and behaviours that deviate from a set or expected course or do not fit within the range of normal behaviour. “Complexity theory…suggests that it might be fruitful to pay greater attention to outliers because they might be a source of new structural arrangements and patterns of behavior” (p. 676).

In understanding unexpected events, we must be cognizant of the tendency to normalize outliers and rogue events. Like the faculty member who scores very high or very low on a teaching evaluation, we might look at our faculty grade report experience and ask why some people did respond on time. We noticed that some of these people had more personal relationships with our program, but this may not have been the case. A faculty member that was flexible to our request had the time or understood the rationale of the request may have contributed to the shape of the outcome.

In other cases, when a change of routine is required, looking to outliers for creativity and improvisation can help illuminate how and where the system is more or less adaptive. Positive Deviance, for example, focuses on learning from anomalies that, despite the odds working against agents in a given environment, are highly successful (Lapping, et. al, 2002). PD was developed from Sternin’s (2005) observation “in every group there are certain individuals whose uncommon but demonstrably successful practices or behaviours enable them to find better solutions than their neighbours or colleagues who have access to exactly the same resources” (paragraph 3). These ‘n=1’ cases may provide insight into how agents in the system have become more or less adaptive to a changing environment.

Proposition 4: Look at dynamics, processes, and patterns across levels

The ways in which people are intended to interact with each other can be found in formal documents and policies outlining role expectations, however, it is the de facto interactions that are the key indicators of system dynamics. The shadow subsystem, or the informal interactions taking place between agents, demonstrates the self-organizing and emergent properties that come from “spontaneously occurring organizational events, structures, processes, groups, and leadership that occur outside of officially sanctioned channels” (Goldstein, 1999, p. 65). Unplanned meetings in the hallways, social cliques, rumour mills, improvised action and life outside of work influence how we think, feel, and communicate with each other. Anderson brought attention to social network analysis as a method for assessing the density and intensity of information flows in a system, the decentralization/centralization of those flows and how nodes in the network can facilitate or block the diffusion of information throughout the network. Because CASs are dynamic, we should look to how these networks change and respond to challenges and uncertain events.

Seeing patterns helps to understand what processes are keeping the system coherent in a particular state or are constraining new behaviours from taking foot. Because of nesting and self-similarity, the patterns revealed by network analyses may be related to patterns occurring at other levels of the system. The grade report responses from faculty may be linked to broader patterns that concern pressures to teach and publish more in the same amount of time. Or perhaps self-perceptions of academic autonomy that suggest doing things at one’s convenience. We are not sure to the extent these patterns exist within our institution, but our point is that if we were interested in enforcing accountability and quality mechanisms without seeing some of the broader patterns, we would only succeed at frustrating faculty.

Conversely, there may be different patterns occurring within or between groups and networks. Looking for overall behaviours neglects the possibility that different agents may, because of their location or connection within a network (i.e. in a ‘clique’ or as a ‘loner’), are subject to or behave in patterns that differ from other agents in the very same network. Female faculty members, for example, need the flexibility to take maternity leave. If teaching and publishing expectations were so high or inflexible as to not accommodate this need, male faculty members might appear to be more compliant with the expectations.

Proposition 5: Observe how the role of the observer is affecting the system
Because of the coevolutionary nature of complex adaptive systems, the role of the observer changes over time as a result of the fact that the system changes, and the system changes as a result of the observer’s presence. Observing these coevolutionary changes is a rich opportunity for gaining insights into system dynamics. (Anderson et. al., 2005, p. 680)

By accepting that we affect and are affected by all sorts of interactions and patterns, Anderson stated that the reflexive and interactive role of an observing agent be used to enhance an understanding of system dynamics. In an attempt to accumulate important information about the behaviour of the system, working to understand the dynamics of the system has the potential to change interactions.

To elucidate, consider the findings from the British Medical Association in 2002 that spoke to the consequences of government-mandated wait-time targets for hip and knee replacements and emergency room visits in Britain’s NHS. A BMA survey showed that hospital workers went to extraordinary lengths to deceive Health Department auditors that targets were being met (BMA, 2002). A number of observations were cited:

•Over half of the audited sites brought in temporary staff until the audit was over.
•One quarter had staff work double or extended shifts.
•Patients were kept in ambulances because ‘waiting time’ does not start until the patient arrives in the department.
•Patients on trolleys were deemed ‘admitted’ despite having no access to food or hygiene facilities.
•Patients were placed on reserve waiting lists so they would not appear on the official waiting list.
•Patients who were near the waiting time target limit were admitted over patients with greater need.

Setting targets is good for trying to improve and measure quality, however, when they come with “a threat of penalties and punishment for those who fail to achieve them [they] make honest people dishonest” (Bogle, 2003). In an educational context, this looks a lot like ‘teaching to the test’. The lessons learned from the NHS experience resonate with our attempts to understand how standards-based efforts to improve education can undermine the desire to make education better.

Conclusion

We must be cognizant of the paradox of our agency; we are constrained but also enabled by our history, local conditions and knowledge. As agents in CASs, we are faced with trying to sort through a multitude of interactions within and beyond a system constantly undergoing change. In an attempt to globally compensate for our local perspective, we end up in a situation in which “too many variables are cast into the melting pot of a system on the assumption that they should be in there and often regardless of whether they are relevant or not” (Pryor & Bright, 2003, p. 125). Pryor & Bright (2003) suggest that this is a consequence of attempting to deal with complexity:

Of course we would like to be able to restrict the range of relevant variables to a small number that could be intensively investigated. But this is to wish for convenience not reality…As for practical relevance, our view is that it is better to start off being over-inclusive and then attempt to rule out some influences on the basis of empirical data (p. 125)

We should confront the complexity of our problems head on and attempt to differentiate between issues in terms of simple, complicated and complex systems. Issues and problems best defined as simple or complicated systems are by no means easy to address, but their functioning is amenable to direct controls and prediction. If we are interested in knowing how institutions are implementing best practices or what evidence is appropriate in terms of an education that is accountable or of high quality, we should be prepared to think about how the complexity of human interaction influences how outcomes come about and how we might measure them.

The study of CAS gives us a language for thinking about and discussing why efforts to change education produce outcomes that seldom match with our expectations. Seeing change as a non-linear process affected by agent interpretation at the local level helps us to see that it is working within these local interactions that desired change at more global levels may emerge.

Acknowledgements and Correspondence

We would like to thank Itay Keshet, Malcolm Richmon, Denise O’Connor, and Tamsin Haggis for their revisions and comments and making the process exciting. This work was made possible by a Canadian Social Sciences and Humanities Research Council (SSHRC) CGS grant. Correspondence may be sent to Sean.Park@learnlink.mcmaster.ca

References

Anderson, C., & Franks, N. (2001). Teams in Animal Societies. Behavioural Ecology,
12(5), pp. 534-540.

Anderson, R., Crabtree, B., Steele, D., & McDaniel, R. Jr. (May, 2005). Case study
research: the view from complexity science. Qualitative Health Research, 15(5), pp. 669-685.

Axelrod, R. & Cohen, M.D. (1999). Harnessing complexity: Organizational
implications of a scientific frontier. New York: The Free Press.

Bogle, I. (30 June 2003). Speech from the Chairman of Council, British Medical
Association. Retrieved January 2nd, 2006 at
http://www.bma.org.uk/ap.nsf/Content/ARM03chcouncil

British Medical Association. (2002). Waits and Measures. Improving emergency
care for today’s patients. London: BMA.

Buckley, W. (1998). Society – A Complex Adaptive System. Amsterdam: Gordon and
Breach Publishers.

Byrne, D. (1998). Complexity Theory and the Social Sciences. New York: Routledge.

Capra, F. (1996). The web of life. New York: Anchor Doubleday.

Capra, F. (2002). The hidden connections. New York: Anchor.

Casti, J. (1994). Complexification: Explaining a paradoxical world through the
science of surprise. New York: Harper Perennial Press.

Chickering, A. & Gamson, Z. (March 1987). Seven Principles for Good Practice in
Undergraduate Education. AAHE Bulletin. Volume 3

Cilliers, P. (1998). Complexity and postmodernism: Understanding complex systems.
New York: Routledge.

Clippinger, J.H. III (1999). Order from the bottom up: Complex adaptive systems and
their management. In J.H. Clippinger (Ed.), The biology of business: Decoding the natural laws of enterprise (pp. 1-30). San Francisco: Jossey-Bass Publishers.

Cutright, M. ed. (2001). Chaos theory and higher education: Leadership, planning
and policy. New York: P. Lang.

Davis, B. & Simmt, E. (2003). Understanding Learning Systems: Mathematics
Education and Complexity Science. Journal for Research in Mathematics Education. Vol. 34 (2), 137-167.

Davis, B. & Sumara, D. (1997). Cognition, complexity, and teacher education.
Harvard Educational Review. Vol. 67 (1), Wilson Education Abstracts. p. 105

Dooley, K. (1997). A Complex Adaptive Systems model of organizational change.
Nonlinear Dynamics, Psychology, & Life Sciences. 1(1), 69-97.

Egan, G. (1994). Working the Shadow Side: A Guide to Positive Behind-The-Scenes Management. San Francisco: Jossey-Bass.

Finnie, R., & Usher, A. (2005). Measuring the Quality of Post-Secondary Education:
Concepts, Current Practices and a Strategic Plan. Toronto: Canadian Policy Research Networks.

Gell-Mann, M. (1994). The quark and the jaguar: Adventures in the simple and the
complex. New York: W.H. Freeman and Company.

Gleick, J. (1987). Chaos: Making a new science. New York: Penguin Books.

Goldstein, J. (1999). Emergence as a construct: History and issues. Emergence-
Journal of Complexity Issues in Organizations and Management, 1(1), 49-72.

Holland, J. (1995). Hidden order: How adaptation builds complexity. Reading, MA:
Perseus Books.

Homer-Dixon, T. (2001). The Ingenuity Gap. Toronto, ON: Vintage Canada.

Kauffman, S. (1993). The origins of order: Self-organization and selection in
evolution. New York: Oxford University Press.

Kauffman, S. (1995). At home in the universe. New York: Oxford University Press

Kelly, S, & Allison, M. (1999). The complexity advantage. New York: McGraw-Hill.

Lee, M. E. (1997). From enlightenment to chaos toward nonmodern social theory.
In R.A. Eve, S. Horsfall, & M. E. Lee (Eds.), Chaos, complexity and sociology: Myths, models and theories (pp. 15-29). Thousand Oaks, CA: Sage.

Lewin, R. (1992). Complexity: Life at the edge of chaos. New York: MacMillan.

Lorenz, E. (1993). The Essence of Chaos. Seattle, Wash.: University of Washington
Press.

Mandelbrot, B. (1983). Fractal Geometry of Nature. New York: Freeman.

Marion, R. (1999). The Edge of Organization: Chaos and Complexity Theories of
Formal Social Systems. Thousand Oaks, CA: Sage Publications Inc.

McPherson, E. (1995). Chaos in the curriculum. Journal of Curriculum Studies,
27(3), 263-279.

McKelvey, B. (1999). Self-organization, complexity catastrophe, and microstate
models at the edge of chaos. In J.A.C. Baum and B. McKelvey (Eds.), Variations In organization science (pp. 297-307). Thousand Oaks, CA: Sage.

Pascale, R., Millemann, M., & Gioja, L. (2002). Surfing the edge of chaos: The laws
of nature and the new laws of business. New York: Crown Books.

Prigogine, I. & Stengers, I. (1984). Order out of chaos. Boulder, CO: New Science.

Pryor, R. & Bright, J. (August, 2003). Order and chaos: a twenty-first century
formulation of careers. Australian Journal of Psychology, 55(2), 121-128.

Sanders, T. I., & McCabe, J.A. (October, 2003). The Use of Complexity Science: A
Report to the U.S. Department of Education. Washington, DC: Washington Centre for Complexity and Public Policy.

Seeley, T. (1995). The Wisdom of the Hive: The Social Physiology of Honey Bee
Colonies. Cambridge, MA: Harvard University Press

Smart, J. (2003). Organizational Effectiveness of 2-Year Colleges: The Centrality of
Cultural and Leadership Complexity. Research in Higher Education, Vol. 44, No. 6, pp. 673-703.

Stacey, R. (1996). Complexity and creativity in organizations. San-Francisco: Berrett-Koehler.

Stacey, R. (2001). Complex responsive processes in organizations. New York:
Routledge.

Sternin, J. (2005). From the inside out: Uncovering solutions to intractable problems
through positive deviance. Retrieved October 31st, 2005 from http://www.leadingedgeseminars.org/sternin.htm

Waldrop, M. (1992). Complexity: The emerging science at the edge of order and
chaos. London: Penguin Books.

Weick, K. (1979). The social psychology of organizing (2nd ed.). New York:
McGraw-Hill.

Weick, K. (1996). Prepare your organization to fight fires. Harvard Business Review,
74, 143-148.

Wheatley, M. (1992). Leadership and the New Science. San Francisco, CA: Berrett-
Koehler Publishers.

Wolfram, S. (2002). A New Kind of Science. Champaign, IL: Wolfram Media.

Zimmerman, B. (2002). It is Time to Differentiate Voluntary Sector Education: A
Discussion Paper for the National Learning Initiative for the Voluntary Sector Project. Retrieved on March 6th, 2005 from http://www.change-ability.ca/NVO.doc

Zimmerman, B., Lindberg, C. & Plsek, P. (1998). Edgeware. Irving, TX: VHA Inc.































Table 1. Simple, Complicated and Complex

System Type Educational Issue/Domain Being Assessed
Simple Operation of most physical systems (i.e. computer networks)
Routine procedures (i.e. accounting, registration, scheduling)
Complicated Integration of physical systems and bureaucratic functions
Proficiency (skills and knowledge in one context)
Complex Culture and climate (i.e. collaborative, innovative, interdisciplinary)
Mastery (application of skills and knowledge across contexts)
Success of students beyond the institution
Changes over time