Over-simplifying in mentoring programmes is a risky business, especially in evaluation, warns guest columnist, Bob Garvey. The ‘realist evaluation’ approach accesses mentoring’s many layers
Within organisations, there are many competing demands on managers – organisations are complex places. But, in complexity, managers, for understandable reasons, tend to try and simplify. It’s probably best summed up by the ‘keep it simple, stupid’ slogan we hear so often in organisations.
When an organisation is developing a mentoring programme, it’s tempting to simplify, control, ignore or smooth over social factors. The risks of this approach to a mentoring scheme (an inherently complex process) are high.
Evaluation is generally recommended as a core condition of any scheme. There are many ways to evaluate but most, in the ‘keep it simple’ tradition, evaluate outcomes. Or when other factors are evaluated, they get downplayed as soft, anecdotal or subjective. A company director once said to me in a discussion on evaluation that ‘people might lie’. I wonder what sort of culture he’d helped create where lying was the norm?
Fairly recently, in coaching programmes, there was a fervent scramble to employ return on investment processes. Fortunately, this approach is now widely discredited, mainly because it professed objectivity and precision while basing the whole approach on subjective guesswork and a dubious attribution of benefits in monetary terms.
A key question is, ‘what is the truth about the programme?’ There could, of course be many answers. Recently, I watched that classic courtroom drama film Twelve Angry Men (1957). It demonstrates that truth can be a subjective process in itself!
In my view, evaluation needs to show what happens, how it happens and why, so that the complexities of the programme can be understood, its impact recognised and processes improved. Enter realist evaluation.
Realist evaluation offers the potential to shed light on the full picture of change in a mentoring programme. There’s something here for coaching programmes too. It’s about ‘what works’ and that has appeal for many managers, perhaps more so than ‘keep it simple’.
First, mentoring is not a kind of recipe. It’s a complex interaction where change comes about through all those who are involved. The design of the programme should create a facilitating environment in which this change can occur. Mentoring happens through personal and interpersonal engagement in developmental relationships and purposeful conversations.
The outcomes, in the main, are individually defined and therefore have meaning to those individuals. Each relationship is unique but, at the same time there are aspects that link to other mentoring relationships across the scheme and to the rest of the organisation – mentoring creates ripples. Realist evaluation is an attempt to access and recognise these layers.
Realist evaluation is a theory-driven approach – and before everyone groans at the thought of theory, remember Lewin’s (1951) comment that “there is nothing as practical as a good theory”. There is theory behind every practice in organisations even though some may deny it.
Pawson and Tilley (1997) were probably the first to make the case for realist evaluation. It’s about the ‘black box’ in mentoring and it looks at how all the inputs convert to the various outcomes. So what are the steps?
Step 1
Establish the theory of the programme.
What is it supposed to do? This may be some kind of ‘theory of change’ framework that attempts to link intent with the outcomes. The following example is taken from an NHS scheme (see Figure 1).
The theory of the scheme:
Mentoring for mentees offers the potential for the following:
- Improved reflection skills
- Support for dealing with specific problems
- Developing strategies for dealing with and resolving:
– crises in professional life
– change in ways of thinking and acting
– changes in direction
- Confidence building in decision making
- Improved self-worth and job satisfaction.
Mentoring for mentors offers the potential for the following:
- Increased motivation and job satisfaction
- Satisfaction for playing a role in developing talent
- Improved relationships with patients, colleagues and family members
- Improved problem-solving abilities
- Improved leadership abilities.
Mentoring for the organisation offers the potential for the following:
- Improved retention
- Improved communication
- Dissemination of values
- Support for recruitment and induction processes and reduced associated costs
- Improved productivity
- Advanced minority groups
- Improved policy implementation.
- Mentoring for patients offers the following:
- Improved health outcomes and a more positive outlook.
Step 2
Consider the three interacting variables:
- What is the context?
- What are the mechanisms?
- What are the outcomes?
And create a logic chain that asks:
What is it about X (context) that gives rise to a causal pathway (mechanism) which leads to Y (outcome)?
Step 3
Formulate appropriate questions to gather data to support the theoretical outcomes and your logic chain and create appropriate data collecting tools, ie, survey, interviews, satisfaction questionnaires, focus groups.
Step 4
Relate the data gathered to the theoretical outcomes and your logic chain, then write your report.
In this way, the realist evaluation process helps identify the parts played in the scheme by the various participants and how these happen.
Realist evaluation is about appreciating the patterns in the linked elements, through contexts, mechanisms and outcomes. Inevitably this is a dynamic process because change is, in itself, dynamic and having a good theory of change within any scheme creates good mentoring practice.