ClickGuard: An Experimental Field Study of Phishing Simulation Training Effectiveness in an Organizational Setting
DOI:
https://doi.org/10.70112/ajcst-2026.15.2.4474Keywords:
Phishing Simulation, Security Awareness Training, Field Experiment, Click-Through Rate, Reporting Rate, Social Engineering, Authority Cues, Organizational Security, Cybercrime, Embedded TrainingAbstract
Simulated phishing campaigns with embedded training are widely used in organizational cybersecurity programs, yet evidence regarding their long-term effectiveness remains mixed. Questions also persist regarding their ability to address evolving phishing techniques and differing patterns of workforce susceptibility. This study evaluates the effectiveness of embedded micro-training and video-based interventions in reducing phishing susceptibility and improving reporting behavior. This study evaluates whether a combined embedded micro-training and video module intervention produces significantly greater reductions in phishing click-through rates (CTR) and improvements in phishing reporting rates compared with standard security policy reminders delivered to a control group. A secondary objective is to determine whether these effects differ across three message themes exploiting authority, scarcity, and generic delivery cues, enabling a theme-specific assessment of training effectiveness that aggregate CTR figures cannot provide. An 8-week randomized field experiment was conducted within a mid-sized organization of 612 employees. Employees were stratified by department and randomly assigned to a training group (n = 306) or control group (n = 306). Three simulated phishing campaigns were delivered at Weeks 1, 4, and 8, each using a distinct social engineering theme. Training group members who clicked a simulated phishing link received immediate embedded micro-training plus a 10-minute structured video module. Control group members received only the standard annual security policy reminder. The training group's CTR fell from 28.3 percent at baseline to 14.7 percent post-intervention, compared with a decline from 27.1 percent to 22.4 percent in the control group (group-by-campaign interaction p < 0.001). Reporting rate increased from 9.8 percent to 31.2 percent in the training group versus 10.5 percent to 14.1 percent in the control group. Median time to report fell from 142 minutes to 47 minutes in the training group. Authority-themed messages showed substantially smaller training-related improvements than generic or scarcity-themed messages, with the training group's authority-message CTR remaining 22.6 percent at Week 4 versus 25.8 percent in the control group. Combined embedded micro-training and structured video delivery produce statistically and operationally significant reductions in phishing CTR and improvements in reporting behavior. These effects are theme-dependent: authority-based social engineering is more resistant to brief training interventions and requires supplementary role-specific and technical defenses, including DNS and IP filtering. Sustained organizational investment in cybersecurity education and attention to emerging AI-generated phishing threats are prerequisites for maintaining training effectiveness over time.
References
[1] D. Hillman, Y. Harel, and E. Toch, "Evaluating organizational phishing awareness training on an enterprise scale," Comput. Secur., vol. 132, p. 103364, 2023, doi: 10.1016/j.cose.2023.103364.
[2] G. Ho, A. Mirian, E. Luo, et al., "Understanding the efficacy of phishing training in practice," in Proc. IEEE Symp. Secur. Privacy (SP), 2025, pp. 76–95, doi: 10.1109/SP61157.2025.00076.
[3] A. Muhammad Auwal, "Adversarial threat modeling in generative AI: A systematic mapping of attack vectors to defense mechanisms," AI Ethics, vol. 6, p. 284, 2026, doi: 10.1007/s43681-026-01154-8.
[4] M. S. Islam, M. Sajjad, M. M. Hasan, and M. S. I. Mazumder, "Phishing attack detecting system using DNS and IP filtering," Asian J. Comput. Sci. Technol., vol. 12, no. 1, pp. 16–20, 2023. [Online]. Available: https://doi.org/10.51983/ajcst-2023.12.1.3552.
[5] A. M. Auwal and S. Lazarus, "Experiences of local victims of Yahoo Boys' socio-economic cybercrimes in Nigeria," Discov. Psychol., vol. 5, art. 161, 2025, doi: 10.1007/s44202-025-00479-5.
[6] M. Auwal and S. Lazarus, "Sociological and criminological research of victimization issues: Preliminary stage and new sphere of cybercrime categorization," J. Digit. Technol. Law, vol. 2, no. 4, pp. 915–942, 2024. [Online]. Available: https://doi.org/10.21202/jdtl.2024.44.
[7] O. O. Blaise, I. Aaron, U. Alfred, and A. Amusa, "Evaluating the ethical frameworks of information security professionals: A comparative analysis," Asian J. Comput. Sci. Technol., vol. 13, no. 2, pp. 61–66, Nov. 2024. [Online]. Available: https://doi.org/10.70112/ajcst-2024.13.2.4289.
[8] E. J. Williams, J. Hinds, and A. N. Joinson, "Exploring susceptibility to phishing in the workplace," Int. J. Hum. Comput. Stud., vol. 120, pp. 1–13, 2018, doi: 10.1016/j.ijhcs.2018.06.004.
[9] L. E. Cohen and M. Felson, "Social change and crime rate trends: A routine activity approach," Am. Sociol. Rev., vol. 44, no. 4, pp. 588–608, 1979.
[10] N. Christie, "The ideal victim," in From Crime Policy to Victim Policy, E. A. Fattah, Ed. London: Palgrave Macmillan, 1986, pp. 17–30.
[11] A. M. Auwal, "Directory brute-forcing and artifact exposure: Qualitative insight into underestimated threats to web application security," Asian J. Eng. Appl. Technol., vol. 14, no. 1, pp. 28–32, 2025, doi: 10.70112/ajeat-2025.14.1.4296.
[12] M. Auwal and J. Olaoluwa Abiodun, "A sectoral study of cyber threats and vulnerability management in resource-constrained Nigerian SMEs," Asian J. Eng. Appl. Technol., vol. 14, no. 1, pp. 33–38, 2025, doi: 10.70112/ajeat-2025.14.1.4297.
[13] M. Auwal, "Autonomy versus algorithm: A replication study of student perspectives on AI ethical boundaries," Int. J. Educ. Technol. High. Educ., vol. 22, art. 72, 2025, doi: 10.1186/s41239-025-00570-w.
[14] J. Wang, Y. Li, and H. R. Rao, "Testing individual differences in phishing vulnerability within an integrated information processing model," Decis. Support Syst., vol. 51, no. 3, pp. 576–586, 2011, doi: 10.1016/j.dss.2011.03.002.
[15] R. Chen, J. Gaia, and H. R. Rao, "An examination of the effect of recent phishing encounters on phishing susceptibility," Decis. Support Syst., vol. 133, p. 113287, 2020, doi: 10.1016/j.dss.2020.113287.
[16] F. Carroll, J. Adejobi, and R. Montasari, "How good are we at detecting a phishing attack? Investigating the evolving phishing attack email and why it continues to successfully deceive society," SN Comput. Sci., vol. 3, art. 170, 2022, doi: 10.1007/s42979-022-01069-1.
[17] A. M. Auwal, "Smart medical IoT security vulnerabilities: Real-time MITM attack analysis, lightweight encryption implementation, and practitioner perceptions in underdeveloped Nigerian healthcare systems," arXiv preprint, 2025, doi: 10.48550/arXiv.2510.09629.
[18] Islam, M. M. Rashid, and F. Othman, "Identifying personality traits associated with phishing susceptibility," Secur. J., vol. 38, art. 18, 2025, doi: 10.1057/s41284-025-00466-4.
[19] K. A. Y. Yaseen, "Importance of cybersecurity in the higher education sector," Asian J. Comput. Sci. Technol., vol. 11, no. 2, pp. 20–24, 2022. [Online]. Available: https://doi.org/10.51983/ajcst-2022.11.2.3448.
[20] S. Ravichandran and K. L. N. Rao, "Design and development of an advancing web information stockpiling for engraved ontology in user contours," Asian J. Comput. Sci. Technol., vol. 11, no. 2, pp. 11–15, 2022. [Online]. Available: https://doi.org/10.51983/ajcst-2022.11.2.3379.
[21] A. M. Auwal and S. Shehu, "Measuring the impact of embedded phishing training on ransomware-lure click behavior: A field study across three organizations," Asian J. Sci. Appl. Technol., vol. 15, no. 1, pp. 47–54, 2026, doi: 10.70112/ajsat-2026.15.1.4350.
[22] A. M. Auwal and S. Shehu, "Psychological, situational, and demographic predictors of susceptibility to ransomware social-engineering lures: A survey study," NeuroData, vol. 3, 2026, doi: 10.63688/neurodata2026305.
[23] A. M. Auwal, "Institutional typology of cybercrime enforcement in Nigeria using EFCC conviction records," Discov. Data, vol. 4, art. 19, 2026, doi: 10.1007/s44248-026-00115-0.
[24] A. M. Auwal, "Invisible intrusions, internal threats, and ethical vulnerabilities in Nigeria's Internet of Things," Discov. Internet Things, vol. 5, art. 128, 2025, doi: 10.1007/s43926-025-00251-0.
[25] A. M. Auwal, "IoT integration in telemedicine: Investigating remote patient monitoring and data transmission reliability," Research Square preprint, 2025, doi: 10.21203/rs.3.rs-3419693/v1.
[26] A. M. Auwal, "A cross cultural extension study on privacy fatigue and AI adoption behavioral model," Discov. Educ., 2026, doi: 10.1007/s44217-026-01903-1.
[27] A. M. Auwal, "Adversarial AI vulnerability in secondary education: Threats and cybersecurity competencies," in Secondary Education: Contemporary Issues, Practices, and Future Directions. London, U.K.: IntechOpen, 2026, doi: 10.5772/intechopen.1016684.
[28] A. M. Auwal and S. Shehu, "To pay or not to pay: An organizational survey of ransom payment decision drivers, data exfiltration, and the role of cyber insurance among mid sized organizations," NeuroData, vol. 3, 2026, doi: 10.63688/neurodata2026306.
[29] A. M. Auwal, "Operationalizing cyber attack prediction: A gap-prioritized framework with dataset and model selection guidelines," arXiv preprint arXiv:2606.03386, 2026, doi: 10.48550/arXiv.2606.03386.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Centre for Research and Innovation
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
