Against the backdrop of sweeping changes in the NHS, clients are reporting increased ability to manage organisational change, among other benefits, according to ongoing evaluation, say Sue Mortlock and Alison Carter
The UK’s National Health Service (NHS) has invested significantly in its executive coach register over the past seven years, in order to quality assure the external executive coaches it uses to coach its senior leaders. The rigorous recruitment to the register has been well-documented (Coaching at Work, vol 5, issue 1). What is less well-known is the work undertaken to evaluate the impact of the coaching this register undertakes.
In 2010, the NHS Institute for Innovation and Improvement commissioned the Institute for Employment Studies (IES), to design a comprehensive and evidence-based coaching evaluation methodology.
This commission arose from increasing public scrutiny on public spending and the need to demonstrate return on investment (ROI).
The existing evaluation had a low response rate (around 20 per cent) and failed to provide the evidence needed to demonstrate coaching outcomes. There was also a desire to contribute to empirical research in coaching.
New evaluation process
The full methodology and toolkit comprises five online surveys and one ROI follow-up telephone interview (see box, p37). At the core of the process are three online surveys completed by the client before, just after and six months after coaching. These give us clear information on any changes over time.
For a multi-stakeholder view, two more surveys are usually also sent: one to a third party nominated by the client and the other to the coach. The post-coaching client includes a ‘trigger/screener question’ to see whether a follow-up telephone interview is worthwhile: “Do you feel the coaching has contributed to any positive outcomes which have (or will) result in quantifiable benefits for patients, or quantifiable improvements for the NHS?”
Those answering yes are sometimes interviewed via telephone to elicit further data. Where appropriate, this is written up as a case study and published on the website to show the outcomes from coaching and to inspire others to come forward for coaching.
Collecting evaluation feedback is an ongoing process, but the analysis can be done whenever we choose. Completion of two key programmes in March 2012 provided an ideal opportunity to take stock of progress.
Key findings
Analysis of the data showed the coaching was having an impact in two main areas:
Benefits to the NHS and patients
Behavioural change in individual leaders
Some 79 per cent of those receiving coaching said they believed it had contributed to positive outcomes that have, or will, result in quantifiable benefits for patients or quantifiable improvements to the NHS. The top areas of positive impact reported by participants were:
Organisational effectiveness
Organisational efficiency
Staff engagement
Staff retention
Quality of care
A comparison of client scores before and after the coaching showed increased levels of self-awareness, self-confidence and resilience and ability to manage politically complex situations.
Even more encouraging was the increased perception in the ability to manage organisation change, given the NHS is in the midst of the biggest change in its 60-year history1 and one of the programmes was specifically funded to support the changes.
Alongside this, coaches reported a marked increase in individuals’ understanding of their own development needs and increased readiness to change.
Learnings
On the positive side, our survey response rate from clients and coaches has shot up to around 78 per cent, despite us sending out more surveys.
The ability to compare ratings before and after coaching has allowed for more confidence in our findings.
The methodology and findings have also raised the profile of coaching within the NHS, and given us evidence of the impact of coaching on patient care.
However, not all aspects have worked well, and we need to further refine and streamline parts of the process. A very low number of peers were initially nominated and we had no responses: we need to consider alternative options to involve other stakeholders and ensure the evaluation is not only based on evidence from the coaches and clients.
The evaluation has been resource-intensive to administer and analyse. We need to invest and make better use of technology. It is also critical the evidence generated from the evaluation is shared with coaches and sponsors to inform the wider leadership agenda.
Conclusion
Given the backdrop of continuing economic pressures, there is a clear need to continue to collect information relevant to ROI, such as stories of benefits to staff and patients, and to calculate ROI from coaching programmes wherever the data is available.
The importance of the NHS continuing to evaluate our coaching offers was underlined by a recent report from the Chartered Management Institute.
It found that higher performing organisations evaluate their leadership development provision to a greater degree
than poorer performing ones. Two of their recommendations are of particular relevance to the coaching evaluation methodology.
First is the need to align measures to support the development of hard evidence of ROI, which the current methodology already does. Second is the use of evaluation data to improve leadership development interventions for even greater impact. The challenge, in a system as complex as the NHS, is how we manage this.
Evaluation of the executive coaching was commissioned by the NHS Institute for Innovation and Improvement in 2010 as part of its work to provide support to leaders from the NHS. The executive coaching register as well as coaching support now forms integral parts of the work of the new NHS Leadership Academy, which brings together for the first time, all the national activity supporting leadership development in health and NHS funded services.
More information on coaching in the NHS as well as further case studies on the impact of coaching are on the Academy website: www.leadershipacademy.nhs.uk
Sue Mortlock is senior leadership associate (formerly head of board development) at the NHS Institute for Innovation and Improvement
Alison Carter is principal associate, Institute of Employment Studies
How did coaching help you?
1. It helped me think differently: In a follow-up telephone interview, a GP client explained how ‘thinking differently’ as a result of being coached helped enable change locally. The coaching kept her mind focused on what was important (enacting change well) and how to do it. In particular, it gave her time and a non-judgmental objective space, separate from the organisation, to talk things through. The coaching allowed the GP to challenge and support others to think and behave differently in understanding and preparing for the new commissioning arrangements.
2. It kept me sane: An executive director explained how coaching helped with the practical challenges in implementing the structural changes, transitional arrangements and cost savings required, while also helping to keep him ‘sane’. This was achieved by promoting resilience and personal coping strategies alongside a focus on maintaining a high-performance environment for the benefit of staff and patients. The organisational benefit is that the primary care trust has continued to perform well through the period of change and he has had some role in this achievement. At a macro level, the organisation has delivered £20m in efficiency savings for 2010/11 and has so far achieved a 40 per cent reduction in staff costs.
He says: “As a scientist I can say the links between the coaching and the impressive organisation results are not definite. [But] I do believe the coaching has helped.”
NHS evaluation toolkit: six tools
1. Client pre-coaching survey
(one month before coaching) covers:
Previous coaching experience(s)
Purpose in coaching
Pre-coaching ratings on key areas
Topics clients intend to focus on
Benefits expected
2. Client post-coaching survey
(one month after final coaching session):
Benefits achieved
Whether client plans to continue with coaching
Satisfaction with coach and coaching process
Post-coaching ratings on key areas
Impact on individual and their organisation
Screener question for follow-up telephone interview to capture return on investment data
3. Client follow-up survey (six months after last funded coaching session):
Whether reported benefits maintained
New benefits achieved
Value of coaching.
4. Follow-up client telephone interview
Conducted as soon as possible after client answers ‘yes’ to screener question
5. Peer survey (six months after final coaching session to colleague nominated by the client):
Ratings on key areas
Value of the coaching
Benefits achieved
6. Coach survey (one month after final coaching session):
Ratings on key areas
Volume 7, issue 6