Evaluating the Effectiveness of Information Security Management Systems: An Analysis Framework and Key Metrics

As large scale digitization continues to reform business processes, one critical challenge organizations are currently facing is managing the staggering amount of data flowing. Further, with large datasets comes the added complexity of insuring a cyber secure environment and shielding the information security management system (ISMS) from undesirable manipulations. Today’s drastic rise of cyberattacks urges the need for effective security frameworks to guard against unauthorized access and malicious acts impeding business operations. The latter of which compelled organizations to adopt holistic information security approaches, commonly implemented via ISMS frameworks. Further, to maintain an effective ISMS, ongoing monitoring and measurements are highly required. Considering the aforementioned points, this paper explores how organizations measure the effectiveness of their ISMS focusing on key performance indicators, metrics, and foundational components involved in information security management by categorizing metrics into governance, risk, and incident response as well as determining the maturity level based on ISO alignment, the presence, specificity and automation of KPIs. Based on empirical interviews with eight diverse organizations, the research findings reveal a wide range of maturity among organizations, from those lacking clear defined KPIs to those with sophisticated multi-layered systems. While special attention is paid to incident-response management, companies with a strong ISMS stand out because they use automated and proactive metrics for strategic reporting, whereas companies with a weaker ISMS often do not have organized KPIs and depend on random manual audits. Based on these results, the present work suggests an analysis framework for evaluating ISMS effectiveness. While previous studies have struggled to define clear ISMS measurement practices, this paper aims to provide insights on measurements by identifying the core building blocks of ISMS and revealing how they are evaluated to drive continual ISMS improvement.