Measuring the impact of human service programs

The challenge: developing evaluation frameworks in complex social systems

Evaluation of human service programs is complicated by the fact that social systems are complex. Complex systems demonstrate non-linear behaviour, where the effect of a given input is dependent on other variables. Unanticipated changes can change the state of the system very rapidly. The system may also reorganise in response to the input, resulting in unexpected outcomes. 

These characteristics challenge traditional measurement frameworks. Government policies and programs almost always exist in complex social ecologies that require mature, integrated evaluative frameworks.

Here are some of the frameworks and approaches ThinkPlace takes to measuring the impact of programs in complex social systems.

Complex systems approach

Systems thinking takes a big picture view of a problem, exploring not only the given problem or program but how it fits into a broader context. Using a systems thinking approach helps to avoid missing out on key information and helps us see how our topic fits in with other related topics or systems such as the health or justice system. We use exploratory techniques and models which participants can apply in their own contexts. 

Design thinking 

Design thinking is an approach to problem solving that puts users at the centre of defining the problem and developing solutions. The discipline transforms big systems by helping us understand the deep experiences within them. In design, we continually zoom between the big complex system and empathy for the individuals that make up the system. We diverge to understand and converge to iterate towards preferred futures.

Theory of Change

The Theory of Change is a framework that helps us graphically represent how programs and strategies are expected to align with activities, outcomes and long term impacts. In contrast to traditional program logic approaches, theory of change models explain “How" and “Why" change is expected to come about. It is a critical tool in strategic planning or policy development to identify the current situation, the intended situation and what needs to be done to move from one to the other.

A Theory of Change is the keystone for developing an integrated evaluative framework. It can be developed for any level of intervention (a pilot, a randomised controlled trial, a policy, a strategy or an innovation hub) and is particularly useful in complex systems, where the focus is to continuously learn at different levels of a system. It can help us:

• Develop pilot objectives and activities, and in general help planning implementation.
• Identify the data needs and optimal analysis methods. 
• Clarify underlying assumptions and preconditions that must be achieved to achieve long-term outcomes.
• Capture internal (e.g., values, guiding principles, operating assumptions) as well as the external contextual factors.
• Capture complex efforts involving multiple stakeholders, whole organisations and multiple programs.
• Develop frameworks that are sensitive to changes and adapt in response to emerging issues and to decisions made by partners and other stakeholders.

Behavioural scientific methodology

Randomised Control Trials are considered the “golden standard” in terms of providing evidence on “what works and why”. Where possible, they should be part of an integrated evaluative framework. However, solely relying on RCTs to gather evidence is not always appropriate. To fully evaluate the impact of programs and interventions in complex systems, we take a mixed approach, integrating both quantitative and qualitative evidence, and experimental and quasi-experimental methods.

Contribution analysis

An effective, mixed approach to evaluation of human service programs requires us to consider both qualitative and quantitative measures across the hierarchy of evidence. However, because of the interdependencies within complex systems, causal attribution is often impossible. Instead, we develop contribution stories based on all the available evidence. How much has a particular program of policies contributed to an outcome? A contribution story considers the interplay of a variety of factors and the awareness that we only have limited ability to extract just our program change from the multitude of changes that happen over time. In developing a contribution story, we systematically explore the often complex and messy nature of programs by accounting for the influence of contextual factors and explanations.

Probe, measure and adapt

Because we are dealing with complex systems, we need to continually revisit our Theory of Change and its associated evaluation framework. Regular measurement of critical indicators and testing of assumptions is critical, so if necessary we can adjust. Because of the feedback loops in a complex system, it is also important to keep in mind that impacts evolve over time. A change program is less likely to be effective if it is implemented linearly—from a single ‘big bang’ intervention to a once-off program evaluation. We need to continually measure and adjust if our programs are to achieve the desired effects. 

Effective measurement starts with a Theory of Change that drives the development of an integrated evaluative framework where both qualitative and quantitative evidence play equally important roles. It is only through an iterative approach, where we continually probe, measure and adapt, that we can create an evaluative framework with sufficient validity to measure the impact of programs, policies and interventions in complex social systems.