It Takes a Village to Build Evaluation Capacity!

September 2, 2016 | By Btissam El Hassar

A Comprehensive Approach to Capacity Building

With growing pressure to make informed funding, policy and program decisions, there is also an increasing demand for evaluations. In this new reality, organizations (e.g., not-for-profits, funders, governmental agencies, universities) often struggle to carry out evaluations that meet both external requirements and yield accurate and useful information for internal purposes. Given the barriers faced in conducting and using evaluation, there is an expanding need to build evaluation capacity. However, building capacity to conduct evaluation and use the information gained from it can be a daunting endeavour. Building evaluation capacity takes place in a complex organizational, social, political, and economic environment (Cousins, Goh, Elliott, & Bourgeois, 2014). It also requires time, financial, and human resources, all of which can be scarce. These challenges can be magnified in complex fields such as early childhood development, in which an organization may have multiple programs with multiple funders who have different evaluation requirements.

Building evaluation capacity therefore requires a comprehensive strategy that is not restricted to only developing knowledge and skills at the individual and organizational levels, but which also considers the need for an enabling environment to support useful evaluation. To borrow an old phrase (and change it for our own purposes), it takes a whole village to build evaluation capacity!

While research on evaluation capacity building (ECB) is ongoing, there is overall agreement in the literature on the key individual (e.g., knowledge, skills, attitude toward evaluation) and organizational (e.g., leadership, communication structure, organizational learning) components (Labin, Duffy, Meyers, Wandersman, & Lesesne, 2012). Investing in a broad ECB strategy can be initially demanding, but it is sustainable over time. For example, with high turnover in some organizations, developing skills of one employee might be more costly over time due to training and retraining. An organization would instead benefit more highly from a broad strategy that involves transforming the organizational culture to one that emphasizes evaluative thinking and reflective practice (Taylor-Powell & Boyd, 2008).

What does a comprehensive ECB strategy entail?

Answering this question is not simple because organizations vary in their complexity, their existing capacities, and the context in which they work. As such, there is no one cookie-cutter solution for all. Currently, the Evaluation Capacity Network (ECN) is in the process of developing an ECB strategy based on consultations with diverse stakeholders (e.g., community agencies, funders, researchers, and evaluators). The ECN conducted four consultations across Alberta and developed a survey to understand evaluation capacity needs and potential solutions of stakeholders to inform the ECB strategy. The results of the process will be shared soon.

The strategy is also informed by the literature on the theory and practice of ECB, which highlights the importance of communication, leadership, and organizational culture. These elements influence organizations’ capacity to conduct evaluation and use its results. Organizations can build or improve their communications, adapt their leadership style, and reshape their culture to one that helps them achieve their evaluation goals. Below are two actions that organizations can take toward building evaluation capacity:

  1. Creating opportunities for knowledge sharing and learning: Organizations exist within a fluid environment impacted by economic, social and political changes. This environment influences how an organization functions and highlights the growing need to learn and adjust to these changes. Given that learning in organizations is a social process, it is important to develop processes that support learning based on continued improvement, risk-taking (and not fear), and team and collective performance (rather than an individual’s) (Preskill & Torres, 1999). One such activity would be to gather employees across all levels of an organization on a regular basis to discuss evaluation results and the lessons learned. Creating room for employees and other stakeholders to discuss and reflect on evaluation and other aspects of their work is essential to creating a learning environment (Preskill & Torres, 1999).
  2. Adopting a leadership approach that encourages innovation, and trial and error. Leadership can play an important role in the effectiveness and sustainability of evaluation capacity efforts (King, 2005; Preskill & Boyle, 2008). Leaders have the power to promote innovation and risk-taking through words and actions. Leaders should not only encourage trial and error but also remunerate learning that occurs as a result of failure as well as success. As Thomas Watson, founder of IBM said: “if you want to succeed, double your mistake rate” (cited in Preskill and Torres, 1999, p. 157).

Building evaluation capacity that is sustainable will take the collaboration of different stakeholders over time. By creating an enabling evaluation environment that supports capacity at all levels, stakeholders in fields such as early childhood development will be able to conduct and use evaluation in ways that are more meaningful.

References:
Cousins, J. B., Goh, S. C., Elliott, C. J., & Bourgeois, I. (2014). Framing the Capacity to Do and Use Evaluation. New Directions for Evaluation, 2014(141), 7–23. http://doi.org/10.1002/ev.20076
King, J. A. (2005). A Proposal to Build Evaluation Capacity at the Bunche-Da Vinci Learning Partnership Academy. New Directions for Evaluation, (106), 85–97.
Labin, S. N., Duffy, J. L., Meyers, D. C., Wandersman, A., & Lesesne, C. A. (2012). A Research Synthesis of the Evaluation Capacity Building Literature. Journal of Evaluation, 33(3), 307–338.
Preskill, H., & Boyle, S. (2008). Insights into evaluation capacity building: Motivations, strategies, outcomes, and lessons learned. Journal of Program Evaluation, 23(3), 147–174.
Preskill, H., & Torres, R. T. (1999). Evaluative inquiry for learning in organizations. Thousand Oaks, CA: Sage Publications Inc.

Other Blogs

Promoting Evaluative Thinking in Everyday Practice: Eval Lab 2.0

An interview with Dr. Bethan Kingsley, an Eval Lab facilitator and former ECN postdoctoral fellow

Relieving the Data Burden

A new call to action addressing the the issue of "data burden" faced by community organizations

5 Insights from Taking a Principles-Based Approach to Measurement

Ryan Lacanilao from Alberta Health shares his reflections on principles-based evaluation