An Evaluation of “The Learning Circles Project”
By Shawn Conway
shawnco@ralphthornton.org

Background

Many months before the Learning Circle Project was approved and funded by the National Literacy Secretariat (N.L.S.), a group of literacy practitioners began meeting in response to the Movement for Canadian Literacy’s National Action Plan for Literacy.  The group was concerned about the proposal for a national adult education system and what they felt was the potential for a focus on academic learning in academic environments.  The group called for “inclusive lifelong learning” to be one of the plan’s goals and considered piloting inclusive community learning centres.  But after several meetings the group realized they did not have enough information about the variety of community learning situations that currently exist.  They decided that a more valuable project would be one that explores and sheds light on inclusive community learning in its various forms.  It was in this context of open-ended exploration that the “Lifelong Learning Working Group,” as the group called itself, was formed and the Learning Circle project was conceived.

The Working Group consisted of highly skilled and seasoned literacy practitioners who had observed the trends and fashions of literacy policy for well over two decades.  They defined themselves as a learning circle, which meant, among other things, that they were committed to an emergent, open-ended process of observation, analysis and reflection.  This exploratory group process mirrored what they believed to be the kind of process that occurs in many other learning circles and is, in fact, a defining characteristic of inclusive learning circles.  It was in this context that I was chosen as the Outside Evaluator for the Learning Circles project.

Evaluation Process

Designing the Evaluation Framework

The initial project proposal submitted to the N.L.S. in late 2003 stated that the objectives of the project were:

• To describe models of informal learning that have managed to include participants who have difficulty using written language, or who might not want to use written language to support their learning;
• To explore some of these models in depth;
• To identify best practices from the models;
• To identify benefits to participants;
• To identify benefits to communities;
• To identify ways in which knowledge about informal, inclusive learning can be shared;
• To explore possible links between this kind of learning and adult literacy programs;
• To propose ways in which inclusive lifelong learning can be strengthened across Canada;
• To propose new approaches to lifelong learning.

As I prepared for my first meeting with the Working Group in the fall of 2004, I considered these objectives and I wondered what the group might define as the indicators of success and outcomes for each of the objectives.  For many years, in human services work and education, nationally and internationally, the accepted wisdom has been that a project is legitimate and meaningful to the degree that it has well-defined and pre-defined indicators, benchmarks and outcome statements.  For example, in a project of this kind, we might create a list of indicators that equate success with participants mastering job application forms or reporting that they have more confidence in using written language for their day-to-day tasks.

Soon into my first meeting with the Working Group it became quite clear that the Group members had no intention of creating lists of pre-defined indicators of success and tailoring their research in order to look for certain details and experiences and not others.  In fact, the Group was ambivalent about the whole concept of “evaluation” laden as it is, especially in the literacy field, with histories of testing, benchmarks, employment preparedness, etc. much of which has been counterproductive and sometimes destructive.  At the same time the Group was very committed to deep engagement with and analysis of learning circles, how they work and how they are effective for their members and communities.

While the Group welcomed the idea of developing a process to keep them focussed on their larger goals, they were opposed to a process that restricted or predetermined opportunities for learning circles and their participants to present or speak for themselves.  As one Group member put it, they wanted to “build indicators without anticipating the end.”  They wanted to keep their broad destination in mind but not “filter” people’s experience through predetermined criteria.

The Group also wanted to avoid coming up with a new formula for lifelong learning that could become a new fetish for policy makers.  That is, the Group did not want their work to support a new policy directive whereby learning circles are defined in a particular way and are promoted as a solution to “literacy problems.”

If, as the original proposal stated, they wanted to “provide a view of the possibilities for inclusive lifelong learning” that learning circle models represent, then they would need an evaluation process peculiar to their open-ended, exploratory research process.  The evaluation process would need to allow for, rather than circumscribe, the organic, dynamic process of the Working Group as well as the open-ended emergent lessons and recommendations that follow from the engagement with the learning circles. 

So, instead of a set of indicators and narrow outcomes we designed an analytical framework that focussed on the three levels or areas of outcomes in the original proposal, namely, the resource book, the research process, and policy recommendations.  The Group agreed to address the following questions on a continuing basis:

• Are we doing / have we done what we said we would do?
• What would we have done differently?
• What are we learning?
• So what?  That is, what has changed / will change / should change because of what we have learned? 
• Now what?  (What is left to do?)
• What are the core factors that contribute to the learning of the Working Group?

Although these guiding questions were uncomplicated, they were in keeping with the Group’s adherence to rigorous and honest reflection at each step in the research.  Thus, while the evaluation framework seemed deceptively simple, it provided a useful guide to a group as strong and committed as the Working Group.

Interim Evaluation Meeting

The next stage in the evaluation process occurred after six months of further research.  During this period the four researchers had been working with their respective learning circles and had been meeting or communicating via the telephone or email regarding their work.

In May of 2005, the whole Working Group and the Outside Evaluator met to discuss progress to date.  In a note from the Project Facilitator prior to the meeting, it was stated that the meeting would be an opportunity to “look at how far the narratives and analysis have moved us toward a resource book and recommendations.  Do we have what we need to develop the resource book and recommendations?  What additional pieces do we need?  At this point, what kinds of recommendations do we see coming out of this project?”

By this time considerable work had been done by the researchers.  The Working Group members were very satisfied with the sixteen draft narratives and the six draft analyses that had been written by the researchers.  From my vantage point as Outside Evaluator, I saw that the researchers had already begun to articulate important lessons and suggestions regarding the overarching themes of how learning circles work and what makes them successful for their members and communities.  There were already specific and rich drafts of writing with titles such as “How Do Things change Because of Learning Circles?”, “Inside the Learning Circle:  What Makes It Work?”, and “Literacy and Inclusion." 

It was also clear that the evaluation framework was providing the Working Group, and the researchers more particularly, with what they needed to stay focussed and to assess progress towards the project’s goals.

My role at this meeting was to help facilitate the Group’s reflection on how far along the work was, what was being learned and what else needed to be done.  In general, the Group members responded well to the draft narratives and analyses and considered other questions and issues that the research had spurred.  For example, an interesting process occurred whereby each level of learning circles provided opportunities for reflection.  In its work with the sixteen learning circles, the researchers relied on each other and became a kind of learning circle themselves.  In turn, the larger Working Group was a learning circle that reflected on and provided feedback about what the researchers were learning.  This multi-layered process of reflection was named “analysis-by-discussion” by the Group and it encouraged a broad and exciting fabric of reflection, a sort of proof that learning circles are good vehicles for lifelong learning. 

By the time of this meeting the researchers had also begun work on a website and had started to summarize what they had learned about learning circles.  The lively and provocative “Beginners Guide to Learning Circles” eventually became the “resource book” envisioned in the original proposal.

The Widening the Circle Symposium

The Group’s recognition of the significance of continual reflection and discussion led the Group to question the completeness of its observations.  They agreed that if they were to continue to emphasize an open-ended reflection process, the project, and all those it touched, could benefit from a symposium involving some of the larger group of learning circle participants.  In September, 2005 the “Widening the Circle” symposium was held and seemingly represented a shift in direction.  The Group felt they needed to check their draft conclusions, to reflect and to explore experiences and concepts with a broader circle of people and specifically current learning circle participants.  (See report for more details.)  In this sense, the symposium was less a shift in direction and more a creative event directly in line with the principles and questions guiding the Working Group.

To the Working Group’s credit, the symposium was a highly successful addition to the whole project.  Discussion at the symposium enriched the draft ideas and conclusions of the researchers and Working Group and provided further material to build into the overall analysis and, in turn, the Guide, the final report and recommendations.  As an outside evaluator, I found the decision to hold a symposium an important testimony not only to the integrity of the Working Group’s process but also to the validity of the emergent, open-ended process of learning circles themselves.  The symposium also showed that the evaluation framework was working well:  the Group was continuing to ask open-ended questions and to gather diverse and abundant material about learning circles.

Feedback from Organizations

The last stage in the project’s evaluation was the solicitation of feedback from literacy practitioners across the country.  The Working Group asked interested people and groups the following questions:

1. What are the implications of a learning circles approach to community learning for adult literacy work?
2. What are the implications of this approach for community development work?
3. What are the implications for a community that you are involved with?
4. Do you have any thoughts about what kind of follow-up would be useful?
5. Plus any other thoughts that occur to you.

Here again the questions are broad and open-ended and indicate the researchers’ thoughtfulness in seeking to hear about others’ experiences and knowledge without circumscribing people’s responses.

Project Outcomes

My final meeting with the Working Group in November, 2006 was dedicated to considering the feedback from the organizations and what was achieved overall in the project.  This was also the time for a final reflection on the evaluative framework as a departure from the standard form of evaluation. 

At this meeting the Working Group debated about some of the feedback particularly whether the intent of the project had been made clear and whether readers could see that the audience for the writing was wide open.  Some felt the project writings needed to be a little more explicit regarding these points while others felt that it was important for the project outcomes to throw up more questions and to spur further reflection.  The various documents are not intended to “represent” all learning circles or to offer a strict formula for them.  Instead, the outcomes (the Guide, the report, the narratives and analyses, and the recommendations) are meant to represent a thoughtful exploration.

Perhaps more significant was the discussion by the Group about the absence of Francophone voices in the project.  While the Group did deliver what it promised, namely the French translation of the final report, the Group members recognized that it would have been useful for Francophone voices to have been involved at various stages of the project.  It was noted that the Working Group had made efforts to involve some Francophone groups but that insufficient resources had been available to do more  especially with respect to translation and feedback. 

Regarding the key desired outcomes of the project – how learning circles work, of what value are they, and what can we learn from them as vehicles for lifelong learning – the Learning Circles Project has clearly delivered what it promised.  In particular, the project has provided the following:

• A plain-language Beginner’s Guide to Learning Circles that can be used as a conceptual introduction to learning circles.  As some symposium participants stated, the Guide also recognizes and provides legitimacy for learning circles as an important model for learning;
• A rich collection of narratives that shows the range and variety of learning circles in Aboriginal, rural, and urban settings;
• A collection of highly insightful and unique analyses cum reflections regarding the learning circles model.  Taken together, the Guide, the narratives and analytical writings make up an ample view of how learning circles can work as models of informal learning, and some of their benefits to participants and communities;
• An example, in the form of the symposium, of how an open-ended exploratory process can lead to a unique forum that engages learners in a sort of meta-analysis (analysis-by-discussion) of a learning process in which they are intimately involved;
• An evaluation framework that resists pre-determined ends but that serves as a rigorous guide in helping a group reach a project’s destination;
• A set of recommendations that tentatively suggest how learning circles can contribute to the public good and how they might be supported;
• A strong case for learning circles as effective and inclusive opportunities for lifelong learning and for contributions to building social capital.

The Learning Circles Project has provided an abundance of ideas and material and exceeded in some ways what it set out to do (symposium, website, etc.)  Although its conclusions and recommendations are especially relevant to the literacy field they are arguably as relevant to many other areas of civil society.   In particular, the concepts of open-ended reflection, emergent knowledge and continual exploration of what is meaningful for the individual and the community would be welcome additions in areas as diverse as environmental justice, community health and community development.

In the field of health, for instance, “learning” rather than behaviour modification could be tied more closely to determinants of health and could itself become a determinant of health.  So too in the fields of community development and community economic development “learning” and learning circles could take an important place beside the standard language of “development,” “growth,” and “progress,” terms which flow from the dominant values of the marketplace rather than core community values.  

Though beyond the scope of this project, how learning circles models could be used elsewhere would constitute very valuable social research.  In the meantime, the many lessons and conclusions of the Learning Circles Project will, it is to be hoped, encourage further work and action towards placing lifelong learning, in all its varieties, closer to the centre of how we live.