Liberating Evaluation: Bringing Systems Change to Impact Measurement & Reporting

We’ve been thinking a lot these days about evaluation. 

As context, NAS recently launched The Project: a pilot offering designed for and by leaders of arts and culture organizations committed to the deep work of equity and systems change. We’ve invested countless hours over multiple years in developing this pilot, and now we’re working on how to most meaningfully understand what’s working, what’s not working, and the resulting impact of the pilot so far. 

This got us thinking about evaluation as its own system. While understanding the impact of our work is a deeply fundamental and essential part of engaging in social change, we also know that systems like capitalism and white supremacy have informed norms of evaluation in our field in ways that can undermine our goals of change rather than furthering them. And when we want to change systems, we need to acknowledge the problematic paradigms that exist currently and envision futures beyond them to move towards collectively.  

So the question is, what might systems change look and feel like within the space of measuring and reporting on impact? What might it be like to engage in truly liberatory evaluation practice? 

What follows are some of the answers that are most present for us as we think about these questions, but we want to hear from you too. Feel free to use the comments section below to add your voice: 

• Funders would ensure that organizations – especially those led by and serving people of color – would associate evaluation with learning, not with fear.Evaluation is meant to be a tool for learning – a way to iterate, evolve, and grow. But for many, evaluation has come to be a source of fear, because in traditional funding paradigms, reporting less than ideal outcomes may result in withdrawal of financial resources. While this is true across the board, it’s especially true for organizations led by and serving people of color. Organizations with access to white privilege still have to navigate complicated power paradigms with funders around evaluation, but BIPOC led organizations also face systemic racism that all too often results in fewer financial and other resources. This means that uncovering and reporting on outcomes that don’t align with funder expectations poses a greater existential threat. And fear, of course, leads to risk aversion, and a desire to mask challenges and learnings that would be better addressed and shared. So while we want organizations to be bold in their visions and work for change, and to be transparent in what they’re learning along the way, the system of evaluation that’s currently predominant directly undermines this.Liberatory evaluation would therefore entail upending this paradigm, working as a field to incentivize leaders to take risks and share openly what they learn along the way. And in doing so, we need to ensure that leaders of color and others who’ve been systematically oppressed benefit from the most safety in evaluation – that the learnings they bring to the table from doing the work, asking hard questions, and making bold choices be a reason to invest more in them. In doing so, evaluation could actually do what it’s meant to and serve as a true tool for learning for all, rather than inadvertently serving as another system that perpetuates inequity.
• Metrics and reporting processes would be co-created, equity-centered, and honor the many ways of knowing.Metrics and reporting processes are often dictated hierarchically by funders and organizational leaders. They’re typically connected to conversations happening in boardrooms, far from the people running programs or benefiting from them. This can create a painful disconnect where the people most proximate to the work are asked for often uncompensated labor to deliver on an evaluation and reporting plan they didn’t get to inform.Liberatory evaluation would instead center the wisdom of those with relevant lived experience, and embrace the many different ways there are of learning, understanding, and sharing knowledge. It would mean letting go of top-down dictates and inviting those participating in the processes to get to inform them. Doing this, however, is inextricably linked to valuing the time and effort of all involved – understanding that when we’re asking people to create or fill out forms, to synthesize learnings and write reports, this time comes at a very real cost. Which brings us to…
• All labor relating to evaluation would be valued and compensated, and experts would be brought in as needed.While it’s true that those most proximate to the work hold great wisdom in it, that doesn’t necessarily make them expert evaluators. And because traditional power paradigms value the time and energy of the people at the top of a hierarchy over others, it’s often the folks most closely connected to the work who are the most overstretched, under-resourced, and underpaid. When funders and/or leaders end up far away from the work of evaluation and reporting, they tend to grossly underestimate the amount of time that goes into it. It sounds easy: collect some data and share it. But as folks who do evaluation work know, there are countless hours put into figuring out what questions to ask, how to get answers, following up as needed to ensure participation, synthesizing learnings, and then sharing findings. The people doing the work know how much it takes, but the systems of power and hierarchy can make it hard for them to share just how heavy a load it is, or to be believed when they do.Given that, in liberatory evaluation practice, it’s essential that funders and leaders first understand the skill and labor involved in this work, and then demonstrate that they value it by investing in it. This may look like:
  • • Budgeting for the substantial costs of personnel time that evaluation requires, and adjusting expectations of those personnel such that they have a realistic workload.
    • Offering compensation to those participating in evaluations such that their time is not taken for granted.
    • Exploring when expert support may be needed and providing funding to hire professional evaluators externally or in house.
• Binaries of success and failure would be relegated to the past.When we design and implement programs, it’s important to start from a place of vision, establish goals, and articulate a desired outcome. Once we’ve done this though, it becomes easy to fall into binary thinking: if we achieve our desired outcome we succeed, and if not, we fail. As with all binaries, however, this thinking is deeply limiting and can even create harm. The reality is, programs and the people they serve can be unpredictable, things change, and there’s always a lesson to be learned. The most impactful organizations are the ones that embrace this – that aren’t afraid to evolve and grow and pivot as needed; the ones that ask hard questions, truly listen to the answers, and are willing to change based on what they learn. But in order to do this, we have to let go of attachment to ideas of success and failure.Liberatory evaluation would fully embrace the notion that there is no binary. In releasing attachment to what success “should” look like and fear of failure, we can all become bolder, take more risks, share learnings more freely, and become more adaptable to changing circumstances in real time.
• Bigger would be dissociated from better, and space for deep thinking and rest would be considered essential.Functioning within the system of capitalism, there is often a push to do more, to scale up, to go bigger. Sometimes, this is an aligned choice for organizations, but sometimes it isn’t. There are times when – in service of the change we want to make – we actually need to go smaller, scale back, go deeper, do less. Many of us know this, but again, traditional funding power paradigms have created norms that make it hard to embody this wisdom in our organizations. As a classic example, many foundations will take time off of grantmaking and/or other activities to engage in strategic planning. This demonstrates a clear understanding of the value of deep thinking and planning work, and the fact that it’s challenging to do such work while also conducting business as usual. The norm for nonprofits, however, is that strategic planning needs to happen on top of all regularly scheduled programming, and ideally while also demonstrating growth. It’s rare that organizations ever feel a sense of permission (either from themselves or their funders) to do less.Liberatory evaluation practice would normalize the fact that smaller numbers don’t equate to lesser impact. It would celebrate the choice of a nonprofit to take time off for deep thinking, because meaningful work requires deep thought and intentionality. And it would also honor the fact that sometimes what makes the work most sustainable is taking time off for rest.

As we think about what liberatory evaluation would feel like to us, we also want to share some resources – links to others exploring these questions in meaningful ways, including: 

1. 1. Liberatory Research – “Liberatory research praxis integrates diverse lineages of knowledge to challenge traditional Western methods and promote equitable knowledge creation. Rooted in postcolonial/decolonial studies, Indigenous research methodologies, and Black feminist thought, it redefines research to be more inclusive and reflective of marginalized voices.” 
   2. Equitable Evaluation Initiative – “We are exploring, co-creating, and advancing a new frame for evaluative thinking in philanthropy”
   3. The Weight Of Power: The Role Of Metrics & Evaluation At The Intersection Of Social Justice – “This report, a collaboration between Echoing Green and CCRE, explores the possibilities of greater equity and progress in social impact by shifting power from philanthropic funders to social innovators and the communities they serve.”
   4. BetterEvaluation – “BetterEvaluation is a knowledge platform and global community. Our vision is a world of better evaluation, better decisions, and better results for people and the planet.”
   5. Social Value International – “Social Value International brings together practitioners, organisations and changemakers from around the world with a shared mission to change the way the world accounts for value.”

Audre Lorde teaches us that “the master’s tools will never dismantle the master’s house.” As we collectively work towards systems change, we need to make sure that the tools, practices, and processes we use for evaluating our work are as liberatory as the change we seek to make.