Assessing systems change: 5 learnings from Stone Soup’s work with clients

October 5th 2023
By Leonora Buckland, Stone Soup Principal Consultant

It is hard to be in a philanthropic or development-oriented conversation without the concept of systems change emerging in the room. Systems change is an idea whose time has come. Yet, the definition of a system can be fuzzy and even overwhelming to many people, particularly to those on the ground experiencing the problems. We have also heard philanthropic boards groan at the concept, struggling to understand what it means and worried that tangible ‘results’ are no longer possible. Articulating what systems change is, why it matters and most especially what it is achieving is essential for those engaged in it who must find a way to communicate learnings, successes and failures.

At its heart, it is a paradigm shift whereby there is an explicit acknowledgement that in order to change persistent, deeply-rooted social and environmental issues, there needs to be more than a series of isolated projects and interventions from the public, social or private sectors. Rather, what is called for is a fundamental change in policies, processes, relationships, and power structures as well as deeply held values and norms related to a system. Iconic examples of systems change include the anti-apartheid movement in South Africa or the civil rights movement in the USA which involved complex webs of people, ideas, wills and action to produce change and seed a new society. The decarbonisation of our economies that we are living and breathing at the moment is another example.

Given the uncertainty and sheer complexity of working with a systems change agenda, how can actors know whether their efforts are succeeding and whether they are ‘moving the needle’ on these stubborn problems? This question is shaking the very foundations of the evaluation or impact measurement industry as we know it which has worked with a more linear framework for understanding change. Familiar evaluation tools have been called into question. We are all now engaged in developing new pathways to understand, learn from and ultimately assess or ‘evaluate’ systems change.

Stone Soup has been privileged to learn alongside our pioneering clients in this matter and in the paper we recently published, Systems change: how will we know it is working?, we discuss some of these examples including work with Ashoka Spain, the Daniel and Nina Carasso Foundation and the Demeter Foundation on how to build systems change interventions and assess them. These examples cover different systems: the entrepreneurial ecosystem in Spain, local food systems in Spain and the system faced by disabled people in Portugal. The key conclusions from our work are:

1. Many organisations engaged in systems change may not have done the basic work to place boundaries around the system they are trying to change, understand context and create a coherent vision for what outcomes they are aiming for. Although the social economy can be caught in an organisational imperative to ‘do stuff’, raise money, demonstrate impact, our insight is that it will be time well spent to study and understand the social, political and economic system before intervening. Getting to grips with a system is hard, but there is no easy short-cut. To do this, systems mapping is a great tool. It can be easy to get lost in complexity so this is an art, not a science, where extracting and illuminating the essence of a system and its interconnections is the goal. Clients over time develop confidence in systems mapping and embed important ‘systems mindsets’ and capabilities within their organisations.

    

2. There are different levels of systems change: structural, relational and transformative as described by FSG in the water of systems change framework. We have found this schema to be helpful so that in addition to assessing change at the structural level, which might be easier to track (i.e. a policy change), we also focus on the more transformative dimensions in possible assessments.

    

3.  Organisations can develop ‘sentinel’ indicators which provide early signals for how far a system is changing. However, obtaining strong, mixed methods (by which we mean triangulating quantitative with qualitative data) and comprehensive information relating to these indicators may not be achievable, particularly for organisations that don’t have large evaluation budgets. For those with more limited budgets, we advise beginning with a few indicators closely linked to the intervention and then revising and expanding the number of indicators as the initiative evolves.
   
   
4. Warning: assessing systems change will rarely give clear results as to what can be directly attributable to an organisation’s intervention. We encourage clients to adopt a learning lens, by which we mean to move away from the accountability framing that can be common when measuring impact (i.e. how much did I get for the resources I put in?). The end game is to learn so as to be able to adapt, rather than reporting upwards to prove impact or legitimise the intervention. Identifying learning questions early on is important.

    

5. Engaging a multiplicity of actors and voices is essential to build systems change initiatives as everyone sees the system differently. A participatory approach requires significant time and energy, to create a common language and to enable disparate organisations and sectors to come together.

Stone Soup’s experience of working with clients on systems change is rich and rewarding, with the development of a systems mindset within each organisation creating a type of superpower. This involves shifting from a more structured, linear approach to a greater comfort with experimentation, not-knowing and a slow, up-and-down path, much of which might not be in an organisation’s control. Humility and curiosity seem to be the key values for actors involved in systems change and these carry through to how it can be evaluated or assessed. Celebrating not only the results of systems change, but the process by which systems change occurs, for example whether it is inclusive and locally-led should become the hallmark of systems change assessment