Workshop in partnership with CEIPS on Thurs 12 March at VicHealth.
In evaluation research, a systems approach provides a conceptual vocabulary and toolkit for improving our understanding of the relationships among the social processes that make programs and interventions ‘work’ as a whole.
In the W3 project we used a systems approach to develop a mid-level program theory for peer and community based health promotion. Here is our theory:
peer-and-community-based health promotion programs mediate between two complex adaptive systems, their target communities and policy environment, and the programs need to fulfil four key functions – engagement, adaptation, influence and alignment – in order to remain effective and sustainable in a constantly changing environment.
In early March we held a workshop with our colleagues at the Centre of Excellence in Implementation and Prevention Science (CEIPS) on using a systems approach in evaluation research with community-based health promotion programs.
Seven things we took away from the workshop
- Facilitation is essential
Systems work sometimes involves moments, or indeed, extended periods of work where the outcome isn’t clear… in fact it isn’t clear there will even be an outcome.
At the end of day one in our workshops with practitioners, we often had to tell participants that it’s normal, at that point, for literally none of it to make sense.
We were lucky to be working with programs and people that trusted us.
As Chris Argyris writes in his article ‘Teaching smart people how to learn’, one of the ‘rules’ of professional life is to minimise, hide and avoid any feelings of uncertainty.
One of the contributions we hope to make is a set of practical suggestions on how to facilitate systems work to support participants with the discomfort of uncertainty.
- What are the seeds and bridges?
The group discussion at the workshop reflected on the need to find simple concepts and vocabulary to spark off initial engagement with the systems approach.
Participant Prof Ray Ison from Monash and Open University suggested a simple vocabulary based on the distinction between systemic (pertaining to a system or understood as a whole) and systematic (methodical, as in following a recipe). That distinction is a potential ‘seed’ concept for adopting a systemic perspective.
Another approach looks for conceptual ‘bridges’ between existing approaches and the systems approach. In his presentation, Graham Brown from the W3 project described using the limitations of program logic in order to ‘walk participants up’ to the more abstract conventions of causal loop diagrams.
This involved presenting two program theory diagrams and asking participants to draw out how the programs might interact. To do that, participants needed to draw the community structures and social processes that mediate between the programs.
Previous workshops had produced a lot of flow charts and organisational charts that observed the formal boundaries between programs. Once over that conceptual ‘bridge’, participants started thinking in terms of a prevention system that spanned the communities, the programs and the policy and funding context.
The challenge is to develop pedagogy for systems research and practice, striking a balance between training and facilitation, and engaging on a strengths basis with the intuitive systems thinking some practitioners already use – rather than forcing them to engage as total novices with an expert discourse of systems theory.
- Getting from ‘WTF’ to ‘Aha!’ – key buy-in points for stakeholders
There was a lot of discussion around the challenges of communicating about systems work, particularly to achieve senior management ‘buy-in’ with the approach.
In the presentation we described three key points where, across workshops with quite different programs, participants and decision-makers went from WTF (‘what’s the function?’) to ‘Aha!’ (‘okay, now I get it’). They were:
- When we used coloured lines to highlight a feedback loop on our causal loop diagram and it became clear the whole thing really did work as a system;
- When we were able to show how the system mapping exercise contributed to concrete implications for their practice, such as strategic considerations and quality indicators for evaluation;
- When we provided a much simpler ‘mid-level theory’ diagram of the four key functions that programs need to fulfil in order to be effective and sustainable.
In particular, senior decision-makers ‘bought in’ at point three: they could see, almost at a glance, how the theory we had modelled in the diagram might help them make the case that peer and community based programs are worth funding. It also showed what evaluation and learning activities need to be funded as well, in order for funders to get the full value of a peer based approach for the sector as a whole.
- Doing communication and change management around systems work
Achieving senior management buy-in was described at the workshop as a particular vulnerability of the systems approach, but in fact, this is going to be a challenge for any new approach to research or practice.
Rather than reinventing that particular wheel within the systems approach per se, approaches like Normalisation Process Theory might offer technical guidance and a research framework that are specialised for dissemination and change management.
- Staging the intellectual work of a systems project
At different points in the W3 project, when it would have been better to be communicating with our partners/stakeholders about our findings, we were still wading through the ‘swampy lowlands’ (Schon, 1987) of the systems approach.
Although we started with the ‘snack size’ texts on systems methods, when the methods didn’t work as promised, we needed to backtrack and read up on the history of the systems approach, to sketch in preliminary and then more elaborate theories of systems ontology (nature of reality) and epistemology (what’s knowable).
That led to some long silences while we were figuring things out. When we came to analysing our data it saved our bacon, but it was pretty uncomfortable in the meantime.
One of the big gaps in the literature on systems is how to interpret the textual and visual products of a systems approach. We had four causal loops diagrams and an absolute mass of rich conversational data from the workshops to analyse.
Traditional methods – breaking the data into sub-units, grouping it into themes, comparing and contrasting between categories – were not leading us anywhere.
We were able to refer back to Pawson & Tilley’s argument about accumulation of evidence by identifying common features of evaluations in different contexts. That got us to a much simpler mid-level theory diagram (see Powerpoint) once we asked what are the core functions that a program needs to fulfil in order to be effective.
Seanna Davidson from Monash University called our attention to way the scope of a systems project shifts over time – there are stages where you want a zoom lens and others where you want a wide angle.
As it happens, early on in the W3 project we used the UK Design Council Double Diamond model of the design process in our project planning:
The diagram shows the changing breadth of focus at different points in the project.
In systems research and practice, projects are likely to involve a whole string of these inter-linked ‘opening up, narrowing down’ operations (Wadsworth, 2011).
The ideal time to communicate might be the mid point between narrowing down in one stage and opening up in the next – firstly to check that you’re not foreclosing something important, and secondly, to communicate with stakeholders at moments of relative certainty and decisiveness.
- Which systems methods suit different kinds of problems/purposes?
At the moment we’re trying to avoid talking about ‘systems thinking’, as it suggests we’re talking about a cognitive habit of wholistic perception.
Instead, we’re talking about ‘a systems approach’ as a family of theories, concepts and methods that can be used to understand the relationships among social processes.
If we wanted to talk about a systems methodology, what’s missing at the moment is theoretical guidance on what kinds of purposes and problems are best suited to different systems methods.
Thinking more explicitly about ontology could also add value to some very concrete and practical decisions that need to be made in prevention and health promotion.
For instance, health promotion work targeting nutrition and physical activity in a local government area might want to ask does our target audience have the structure and functions of a community? If it doesn’t, then local planning might need to prioritise strategies that don’t depend on these structures and functions to succeed.
- How could we support people to do ‘a little bit’ of systems as needed?
Some authors argue that a systems perspective will someday replace ‘the dominant paradigm’ in research for policy and program decision-making.
Perhaps as a result, they don’t say much about how systems work might complement and fit alongside projects based on other perspectives.
For the foreseeable future, systems work is going to be one part of a larger mix of perspectives and approaches that inform policy and practice.
It would be helpful to think about how we can communicate our findings without our audience members needing to learn about systems themselves.
The really key takehome from our workshop was to get out there and give it a go – but don’t be afraid to adapt the systems approach to make it work for your problem or purpose, and the setting in which you’re working.
Do seek out advice from people who’ve used systems approaches before, but always keep in mind that you’re using systems to add value to your own project and not the other way around.
At the workshop we asked six small groups to reflect on two questions for fifteen minutes each. The questions were:
- What have been the ‘sticking points’ in your past experiences of projects using the systems approach? (Or, for people who were new to systems, what might be holding you back from trying it?)
- When and how have you needed to ‘break the rules’ of a systems approach in order to get or keep your project moving? (And for people who are new to systems, how might you need to adapt the approach to your local needs?)
When we came back together as a whole group, we reframed the questions.
- Question 1 became: What challenges do we need to address to promote wider uptake of systems approaches?
- Question 2 became: What guidance can we offer practitioners on how to adapt systems approaches for relevance to different problems/settings?
The results follow below.
A1. Challenges to address to improve uptake of systems approaches in health promotion funding, planning, implementation and evaluation
- Working with uncertainty
- Giving practitioners names/vocabulary for intuitive systems thinking
- Establishing shorthand for systems concepts
- System, systemic, systematic (Ray Ison)
- Finding ‘bridging concepts’ for non-systems thinkers
- Different time scales between practice knowledge, evaluation, research
- Challenging systems is uncomfortable
- Defining quality use of the systems approach and indicators of it
- Thinking about who can’t participate (easily or at all) in systems work
- Accounting to communities for use and outcomes of systems approaches
- Demonstrating the value of systems work to decision-makers
- Painting vivid pictures both of and from systems work
- Some health system structures may be inimical to a systems approach
A2. What guidance do we need on adopting and adapting a systems approach to meet local practice and project needs?
- The guidance is already out there – but there were two views on this:
- ‘it’s not taught to the people it should be’
- ‘it’s too rigorous to be picked up and used in practice’
- What are the ‘seeds’ for developing systems knowledge?
- Authorising users of systems methods to define their own level of focus
- How do we pose the quality question (#7 above) to non-systems work
- What systems methods work for which groups
- How can we ‘stage’ the intellectual work of a systems project (see #4)
- Translating our work in communication with non-systems stakeholders
- Change management to achieve organisational buy-in
- Making safe spaces for practitioners to demystify implicit practices
- Using double-loop model of reflection on systems practice (Ray Ison)
- Demonstrating the investment curve with public buy-in to –
- community-based health promotion
- systems approach in (a)
- Knowing how to use ‘a little bit of systems’ when and as needed