Cybersecurity training programs have become a staple of organisational cybersecurity for a number of years now. Companies are dedicated to educating their employees on the risks they are exposed to, particularly because employees often constitute the last line of defence for a variety of cybersecurity threats and frequently engage in risky online behaviours. Despite the prevalence of these training programs, a systematic analysis of their effectiveness has so far not been conducted. For this reason, Dr. Tommy van Steen (Assistant Professor of Cybersecurity Governance), Dr. Bibi van den Berg (Professor of Cybersecurity Governance), and I investigated the effectiveness of cybersecurity training programs through a meta-analysis.

We carried out the meta-analysis based on the search criteria used in a systematic literature review that was published in 2023 and included an updated search based on the same criteria which was carried out in August 2023. Overall, we retrieved 69 studies from the literature, which were then used to determine the overall impact of cybersecurity training on end-users. Studies were categorised according to the study design that was employed, as well as the outcome measures that were used to assess effectiveness. Within the sample, the use of both independent groups as well as repeated measures designs could be observed. When it comes to outcome measures, the majority of studies (48) assessed training effectiveness through precursors of behaviour, such as intentions, knowledge or attitudes. In comparison, 33 studies assessed behaviour either through objective measures such as phishing click rates or made use of participant self-reports.

Effect of training on behaviour

The initial assessment of training effectiveness revealed that, overall, a positive effect of cybersecurity training can be observed, as indicated by the medium-to-large effect size (d = 0.75) that was generated in the analysis. In addition to the overall meta-analysis, we ran several subgroup analyses to assess the effect of training for the conditional factors that were observed in the data, such as study design and outcome measures. These analyses found that while cybersecurity training was highly effective in influencing precursors of behaviour, irrespective of the study design that was used, studies assessing behaviour change using independent groups on average show only a small, non-significant effect (d = 0.36). Overall, these findings show that while training programs are effective in changing knowledge, attitudes and other precursors to behaviour, they are less effective in triggering actual behaviour change, which is the overall end goal of training. Considering the necessity of end-user training to ensure the continued safety of organisations, as well as the high costs often associated with implementing these training programs, this finding is highly relevant.

Training characteristics

In addition to the effect of training on precursors vs. behaviour, we were also interested to assess the impact of various training characteristics on training effectiveness. These moderators included the method that the training was delivered with, the use of more than one training method, as well as the platform (e.g. online vs. offline) and social setting (group vs. individual) in which the training was administered. The analysis showed that while some training methods, such as games or presentations, produce larger effects than other methods (e.g. discussion-based approaches) the overall difference between them is not strong enough to be statistically significant. Similarly, the use of multiple training methods or the platform and social setting of the training did not significantly moderate the effect.

Conclusion

Overall, though we cannot say conclusively why there is such a discrepancy between the effects of training on precursors and behaviour, it is too large to simply be explained by established behavioural theories such as the Theory of Planned Behaviour, which posits that intentions and other predictors are easier to influence than behaviour. We speculate that there is a potential unsuitability of current training methods to address actual behaviour, as there is no differentiation in how certain cybersecurity behaviours are trained. Here, it is not necessarily the case that one method of training is superior over another in general, as shown by the various moderator analyses that were conducted as part of this analysis, but that we need to find methods of training that are suitable for each cybersecurity behaviour we are attempting to address individually. Future research should be concerned with identifying unique characteristics of the different cybersecurity behaviours that end-users engage in and outlining training methods that address these characteristics.