ROI in executive coaching? For the sake of our profession, let’s stop using it, says Willem Jan Hofmans

There is a lack of empirical evidence that coaching delivers the desired results, according to much of the academic literature, prompting me to focus my research for my Doctorate in Business Administration (DBA) on the efficacy of executive/leadership coaching. My conclusions included that ROI evaluations are not only a waste of time, but potentially damaging to the profession, leading me to develop an alternative evaluation framework, which I outline here.

 

Existing research

There are few empirical studies which link leadership coaching to improved results. From a managerial viewpoint the issue has been that the Return on Investment (ROI) of coaching is not easily measurable (AMA, 2008). From an academic perspective, the problem is that “…little guidance exists on how to evaluate this unique leadership development practice” (Ely, Boyce, Nelson, Zaccaro, Hernez-Broome & Whyman, 2010, p585).

De Haan, Duckworth, Birch & Jones (2013) have proposed that outcome results similar to psychotherapy can be assumed. However, from a business perspective, this assumption may be insufficient as tangible results are typically required in this field.

As ROI is a well-known term in business, it can be attractive for coaches to use. Some hold that it is ‘only’ a matter of looking at the (usually) estimated returns from the coaching, eg, more revenues, less costs, increased productivity and dividing these by total costs for the coaching.

Reported ROIs of 545% (McGovern, Lindemann, Vergara, Murphy, Barker & Warrenfeltz, 2001) and 221% (Philips, 2007) are two examples. However, these types of ROI would make many financial advisers feel ashamed.

Whereas the efforts to attempt to calculate ROI for coaching interventions are laudable and, without a doubt, done with good intent, my spontaneous reaction to these is that either there must be something wrong with the assumptions or that the coach charged way too little for the coaching. I believe these types of published ROI actually hurt our profession, not support it.

As part of my doctoral research, I investigated which approaches have been taken to establish the benefits of leadership coaching. How do we know that a leader has become more effective and that the coaching was worth it?

Two of the main approaches described in the coaching effectiveness literature are self-reporting by clients and/or coaches and some longitudinal pre/post-coaching studies. Having attempted to address these in my study, I can confirm it is a lot simpler to just focus on a post-coaching assessment of the engagement by the client. However, apart from being subjective, one could wonder what is really being measured?

 

An Alternative

As an alternative to traditional ROI, I propose the integrated evaluation framework for leadership coaching (Figure 1). It has been derived from findings from the meta-analysis of research projects on the impact of coaching from Ely et al (2010), De Meuse, Dai & Lee (2009) and Theeboom, Beersma & van Vianen (2013), in combination with the work by Kirkpatrick (1996) on evaluating training programmes and Chen (1989) on evaluation of programme theory.

To explain the framework I’ll use the training evaluation model developed by Donald Kirkpatrick (1996). It describes four levels:

• Reaction, a measure of how participants feel about the various aspects of a training programme, including the topic, speaker, schedule, etc – basically a measure of satisfaction
• Learning, a measure of the knowledge acquired, skills improved or attitudes changed due to training
• Behaviour, a measure of the extent to which participants change their on-the-job behaviour because of the training
• Results, which measures the final results that occur due to training, including increased sales, higher productivity, bigger profits, reduced costs, less employee turnover and improved quality.

(Note: the higher the level, the more difficult and expensive it is to assess)

The model lends itself very well to use in coaching. Any well-educated (and certified) coach will check in with their client both during and at the end of a coaching engagement to verify the client’s level of Reaction (read Satisfaction). My doctoral research (Hofmans, 2015) confirmed the significance of client satisfaction in coaching outcomes.

 

Satisfaction

What was even more interesting is that the Satisfaction of the coach, as also reported by Boyce et al (2010) and Baron & Morin (2009), seems to have a much bigger impact on the relationship between the client and the coach, and possibly coaching outcomes, than previously believed.

“Coaches may not be as ‘in tune’ with their clients as they typically assume” (De Haan et al, 2013, p54). So, as a coach, in addition to focusing on your client’s satisfaction, you may even be more conscious of how satisfied you are with your client and the coaching, and what you are going to do with this, to ensure an even better outcome for your client.

As far as Learning is concerned, the suggestion is to use Achievement of Coaching Objectives, also commonly used by coaches, as the key indicator. To be as objective as possible, the recommendation is to not only have the client evaluate this, but also involve the client’s boss. Whereas correlations between the objectives and the leadership dimensions were strong (Hofmans, 2015), the statistical significance of Achievement of Coaching Objectives to be a predictor for improved coaching outcomes was only 93.9%, which is just below the threshold of 95%.

In the literature, Bowles et al (2007), Evers, Brouwers & Tomic (2006) and Orenstein (2006) have documented positive findings as it relates to setting goals in coaching. The recommendation is to define up to three measurable objectives before the coaching starts, and review these at regular intervals and at the end of the engagement with the client and his/her boss as the indicator of Learning.

To measure changes in Behaviour, changes in Leadership Effectiveness, from a 360 instrument (as reported by all evaluators except the client), and Mindfulness, as reported by the client, have been used in the study (Hofmans, 2015) as both a short- and long-term indicator, respectively.

The Leadership Effectiveness for 30 clients increased over the mean coaching of 8.7 months, but the change was not statistically significant. This was possibly influenced by the fact that only 67% of the evaluators were the same between post- and pre-coaching.

Mindfulness of the clients did increase significantly. Given the criticism in the literature of self-reporting, it could be argued that this measure was not objective enough.
One could question how relevant the evaluation of Mindfulness by others would be. The main take-away is that a Leadership Effectiveness measure, as evaluated by others, in combination with a measure of Mindfulness, both before and after coaching can be useful to measure a change in Behaviour, as long as it can be assured that evaluators complete both evaluations.

This brings us to the fourth level, Results. In the study (Hofmans, 2015) a Business Performance Index was used, completed both by the clients and their respective bosses before and after the coaching. Again, there were improvements, but they were not statistically significant. The instrument itself proved to be a bit of a challenge and, if I were to do the research again, I would look for an alternative.

Nevertheless, the proof of concept worked and I would recommend practising coaches to agree upfront with their clients what objective indicators for business results could be and then measure these before and after coaching. Considerations would of course have to be given to other factors that may have contributed. As also mentioned by Kirkpatrick (1996), ideally a follow-up measure should be done six to 12 months later. It should be clear that in this case the sole contribution of the coaching to the results would be even more difficult.

 

Conclusion

A simple ROI calculation for coaching does not exist. Measuring coaching effectiveness is much more complicated. It is also not in the interest of our profession to proudly present and promote huge ROIs. A measured approach as proposed earlier is more reasonable and credible. This will contribute to a profession that is serious about how outcomes are evaluated and reported.

 

• Willem Jan Hofmans has been a leadership coach since 2007. He lives in Switzerland

 

 

 

References

• American Management Association, Coaching: A Global Study of Successful Practices, 2008
• L Baron and L Morin, ‘The coach-coachee relationship in executive coaching: A field study’, in Human Resource Development Quarterly, 20 (1), pp85-105, 2009
• S Bowles, C J L Cunningham, G M De La Rosa and J Picano, ‘Coaching leaders in middle and executive management: Goals, performance, buy-in’, in Leadership & Organization Development Journal, 28(5), pp388-408, 2007 http://bit.ly/2jxgimM
• L A Boyce, R J Jackson and L J Neal, ‘Building successful leadership coaching relationships: Examining impact of matching criteria in a leadership coaching program’, in Journal of Management Development, 29(10), pp914–931, 2010 http://bit.ly/2jxhG8O
• H Chen, ‘The conceptual framework of the theory-driven perspective’, in Evaluation and Program Planning, 12(4), pp391–396, 1989 http://bit.ly/2jWKmEj
• E De Haan, A Duckworth, D Birch and C Jones, ‘Executive coaching outcome research: The contribution of common factors such as relationship, personality match, and self-efficacy’, in Consulting Psychology Journal: Practice and Research, 65(1), pp40–57, 2013 http://bit.ly/2ktvHUc
• K P De Meuse, G Dai and R J Lee, ‘Evaluating the effectiveness of executive coaching: Beyond ROI?’, in Coaching: An International Journal of Theory, Research and Practice, 2(2), pp117–134, 2009 http://bit.ly/2kXbPWs
• K Ely, L A Boyce, J K Nelson, S J Zaccaro, G Hernez-Broome and W Whyman, ‘Evaluating leadership coaching: A review and integrated framework’, in The Leadership Quarterly, 21(4), pp585–599, 2010 http://bit.ly/2jEICyF
• W J G Evers, A Brouwers and W Tomic, ‘A quasi-experimental study on management coaching effectiveness’, in Consulting Psychology Journal: Practice & Research, 58(3), pp174–182, 2006 http://bit.ly/2kT07jO
• W J Hofmans, ‘Effectiveness of leadership coaching – An integrated evaluation framework’, DBA dissertation, Grenoble École de Management, 2015
• D Kirkpatrick, ‘Great ideas revisited’, in Training & Development, 50(1), p54, 1996
• J McGovern, M Lindemann, M Vergara, S Murphy L Barker and R Warrenfeltz, ‘Maximizing the impact of executive coaching: Behavioral change, organizational outcomes, and return on investment’, in The Manchester Review, 6(1), pp1-9, 2001
• R L Orenstein,‘Measuring executive coaching efficacy? The answer was right here all the time’, in Consulting Psychology Journal: Practice & Research, 58(2), pp106–116, 2006 http://bit.ly/2kT4Kuq
• J J Phillips, ‘Measuring the ROI of a coaching intervention, Part 2’, in Performance Improvement, 46(10), pp10-23, 2007 http://bit.ly/2jEQy31
• T Theeboom, B Beersma and A E M van Vianen, ‘Does coaching work? A meta-analysis on the effects of coaching on individual level outcomes in an organizational context’, in The Journal of Positive Psychology, 9(1), pp1–18, 2013