Any time a computer system changes, there is potential that while one area improves, another area is negatively impacted. As a result, it is important to benchmark cybersecurity environments before any change is applied. After an update to a business’s cyber security framework, comparisons can be made to assess if the results of the change are positive.
By the end of this topic, you will understand:
- how to benchmark cyber security in a business
- how to evaluate cyber security change and adjust as needed.
A benchmark is a point in time reference. Sets of values at that time are recorded. When a benchmark is recorded, it can be used to measure other times to compare the same sets of values. In business, some benchmarks become the minimum standards that the business wants to meet. For example, a business may set a benchmark standard to achieve a certain profit amount each month.
Benchmarks are likewise used in cyber security.
The types of benchmarks available for cyber security managers to help analyse how successful the cyber security approach is in their business belong to one of two categories – internal and external.
Internal Benchmarks
Performance data is gathered from across the business and recorded. The data can be referenced at later dates to compare relative progress. Cyber security frameworks assist in describing the metrics (a term for measurements taken) that, depending on what the benchmark measures, provide targets to measure.
Compliance checks also gather metrics that can be used to build benchmarking.
Note
As discussed, measuring attempted and successful cyber-attacks is not a measure of the success of a business’ cyber security. However, if a cyber threat is successful, a significant investigation should be conducted to identify how the business’ cyber security practices failed.
Useful metrics that can be recorded include:
- how many devices and applications are not patched to the latest versions as required by cyber security policy (e.g., within 2 weeks of a patch being released).
- mean times:
- measure the mean time taken to identify a cyber threat (mean time to Detect)
- measure the mean time taken to remedy a cyber threat (mean time to Resolve).
- cyber training – based on results from training assessments that gauge competency and comprehension of staff regarding cyber security
- results from vulnerability scans and penetration tests
Internal benchmarking includes gathering quantitative and qualitative data. Aside from numerical measuring, discussing with staff and management internal benchmarking can indicate changes in awareness and perspective regarding the perception of cyber security in the workplace.
Benchmarks have practical value in evaluating cyber security strategies. Whether a change in the approach to cyber security in a business is sponsored by new policy and procedure, or through the application of new technical capabilities, measuring with useful metrics before and after the change (and continuing to measure), allows comparison points. The points can be directly compared to establish:
- if cyber security is improving technically (by comparing penetration testing for example). The results of earlier penetration tests are compared with current tests to ensure continual closing and mitigation of security threats.
- if cyber security awareness is improved by certain training courses (by performing assessments, before and after training. Knowledge retention can be tested in the later months to identify if refresher training is needed.
Case Study
Technical Benchmarking
Every 6 months, ACE Pty Ltd conducts a penetration test and a full ACE business wide Vulnerability Scan (VS).
1) 18 months ago, the results of the testing were as follows (this set of tests is called 2021AUG within ACE):
Vulnerability Scans:
Identified 5 mobile phones, 2 PCs that had versions of Windows and Android that were out of date.
Penetration Tests:
Penetration tests found that physical security was appropriate – server rooms were secure, mobile devices did not hold sensitive information and mobile users were using MFA. In the event a mobile device was lost, no exposure to hackers was possible. Policy on mobile devices was being followed (not using public Wi-Fi or USB sticks for example). PCs in the business complied with policy and users likewise did not keep business data on PCs (they used the business cloud) and used MFA.
Technical penetration tests found that one exploit remained open – SQL injection was possible from the company website. This was due to a new function on the business website that was added without involving the cyber security team.
(SQL injection is a technique to use SQL queries to extract data beyond the intention of the application – refer to: SQL Injection (w3schools.com)
Minimum Standards Expected
ACE cyber security adopted the following internal benchmark for cyber open threats – open meaning not mitigated when VS and penetration tests are conducted. Vulnerability Scan Benchmark: 1 PC and 1 mobile phone not patched as required Penetration Testing Benchmark: Physical security to be as robust as the results from 2021AUG – 0 exploits discovered.
2)12 months ago, the results of the testing were as follows. This set of tests is called 2022Dec.
Vulnerability Scans:
0 devices were found to contain vulnerabilities
Penetration Testing:
1 exploit was uncovered that was a technical exploit. The webserver had not been patched to the appropriate software release within two weeks of the patch being released. This was due to an oversight by the IT department due to resource shortages.
Benchmark Results: Compared to desired benchmark, the result was good and the one exploit found was explained and remedied.
3) 6 months ago, the results of the testing were as follows. This set of tests is called 2023June
Vulnerability Scans:
12 devices were found to contain vulnerabilities.
Penetration Testing:
4 exploits were uncovered that were technical exploits and physical exploits:
- A server room was not secured
- SQL injection could be performed from a business website
- A buffer overflow exploit was successful on a business web server
- Data base admin usernames and passwords were the default
Benchmark Results:
Compared to the desired benchmark, the result was poor. This was expected as the tests also assessed a recently acquired trucking company’s digital platform. The benchmarking was conducted before the IT systems of the trucking company were integrated to ACE’s systems.
As ACE’s cyber security manager, you had pointed out weaknesses in the webserver earlier to senior management. You circulate a report of the benchmarking results to management and indicate:
- the deviation in the results from those that are expected and typical for ACE
- what the exploits and vulnerabilities mean as well as the timeline and budget necessary to close the gaps.
External Benchmarking
Businesses are regularly surveyed to gather trends and approaches to delivering cyber security. By accessing these reports, a business can compare its approach to that of the wider industry. Where there are differences, a business can examine why its approach varies from that of its peers.
Example
In a survey conducted by Gartner in 2021 (participant numbers were 265 directors of large companies globally), responses indicated: By 2025, 40% of companies will have a dedicated board member working with a cyber security committee.
Cyber security managers look to various reports to gather the performance of how effective the strategies employed in a business. Many are real time (such as intrusion detection and protection software) or regular activities.
Reading
Network Intrusion Software
Real time Intrusion Detection System (IDS) and Intrusion Protection System (IPS) software have two complimentary functions.
IDS continuously looks for suspicious network data that, while permitted by a business’s firewall, contains structures that are indicative of a cyberattack. IPS software, once network packets are identified as suspicious, ‘drops’ or blocks the data and creates a report on the attempt (where did the attack originate from, what was the data and why did the IDS trigger)
For more information refer to: Top 6 Free Network Intrusion Detection Systems (NIDS) Software in 2023 | UpGuard
Activity 1
Benchmarking informs change. Cyber security managers need to assess the impact of changes to the cyber security framework in a business to ensure:
- that the changes are working as intended (e.g., remedying a gap in cyber security)
- that changes are not having unintended consequences
- evaluation of the process that delivered the change (this assesses mean time benchmark indicators such as the Mean Time to Resolve.
No assumptions can be made on the efficacy of a change. Cyber security managers need to be especially aware that significant technical changes (such as an entirely new release of software) often lead to a quick succession of minor patches to close security exploits the major version update introduced.
Benchmarking tools include creating internal KPIs (Key performance indicators), Using CIS and NIST custom benchmarks and Cyber Security Maturity modelling:
Creating KPIs
Creating internal KPIs (Key performance indicators) to compare the historical or current state of cyber security with goals of the organisation regarding cyber security.
A lack of cyber-attacks is not a valuable KPI. It may be that hackers have simply not made an attack. KPIs should reflect continued ‘hardening’ of a business’s digital platforms to close gaps and improve security on a technical and procedural level. Improvements in threat recognition, cyber security training delivery and assessment success and successful testing of response plans. Aside from the useful metrics we previously addressed (such as cyber training retention success), the following KPIs should also be considered:
- Cost per cyber incident – in the event that an attack is attempted, what cost is met by the business
- Security policy compliance – how many incidents are there of non-compliance from across the business in workplace behaviours
- Time to apply security patches – how long does it take from a patch being released to the when the patch is applied.
Ideally, KPIs trend to optimal levels (for example, less non-compliance and faster patch application.
CIS and NIST custom benchmarks
CIS and NIST also provide custom benchmarks for existing businesses as well as frameworks.
CIS offers checklists based on:
- Operating system
- Server software
- Cloud providers
- Mobile devices
- Network devices
- Desktop software.
Compared to the CIS framework itself, these benchmarks provide specific directions on how to deliver the task in the framework. For example, the Windows 10 standalone benchmark indicates that account policies should ensure a minimum password length of 14 characters.
The NIST framework provides benchmark models designed to support the specific needs of a business based on the industry. Called profiles, these models include specific tasks that are geared towards meeting the needs of businesses in that industry (for example, manufacturing). Further, NIST benchmarks for specific threats such as ransomware attacks.
Reading
To find out more about CIS benchmarks and NIST Framework Profiles, follow these links:
- CIS Benchmarks: What are CIS Benchmarks? | IBM
- NIST Framework Profiles: Examples of Framework Profiles | NIST
Cyber Security Maturity (CSM) modelling
CSM provides a holistic assessment of a business at a point in time. It is a snapshot of the level of adoption of cyber security principles found in cyber security frameworks. CSM models are based on the principle that businesses journey from limited cyber security, continuously improve their levels of security optimisation and the ideally become fully ‘cyber optimised’.
Being cyber optimised is when a business always:
- reacts to threats using a full suite of threat detection software without delay
- has policy and processes that follow, as required, an extensive framework (e.g.,NIST/CIS)
- has partners and clients who are cyber security managed
- has staff and stakeholders that always adhere to policy.
Because being ‘cyber optimised’ is an ideal state, making an effort to achieve it provides a target and a direction of travel when undertaking a CSM review. There are several CSM models, and they differ in their approaches, but all rely on a cyber optimised end point.
Explore
For example, the Cyber Security Capability Maturity Model (C2M2) was developed for use by the US energy industry and allows self-evaluation of maturity using a set of downloadable tools: Cybersecurity Capability Maturity Model (C2M2) | Department of Energy
The model is aligned with the NIST cyber security framework. It model works by assessing the characteristics, attributes, and indicators of a business' cyber security using three levels of preparedness for 10 Domains. These domains represent 10 areas of a business that require protection and are delivered by the NIST framework. The 10 domains are:
| Risk Management (RISK) | To operate and maintain a cyber security risk management across the entire business. |
| Identity and Access Management (ACCESS) | Physical and logical access is provided only as needed. |
| Situational Awareness (SITUATION) | Tasks and technology that monitors, analyses and reports hazards. |
| Continuity of Operations (RESPONSE) | Tested procedures and plans in place to recover from identified threats. |
| Third-Party Risk Management (THIRD PARTIES) | Controls and policy in place to secure threats linked to third parties. |
| Workforce Management (WORKFORCE) | A strong culture of cyber security awareness exists, and the workforce is competent to manage critical business assets. |
| Cybersecurity Architecture (ARCHITECTURE) | The policy, processes and technologies that underpin cyber security is sufficient to protect the business. |
| Cybersecurity Program Management (PROGRAM) | The governance and planning of cyber security in a business aligns with business objectives AND protects the businesses digital platforms. |
| Asset, Change, and Configuration Management (ASSET) | Hardware, software and data are managed to meet the risk of cyber-attack and meet organisational objectives. |
| Threat and Vulnerability Management (THREAT) | Threats are identified and managed as needed to protect the business. |
MILS
Mil 0 No practices in the domain are conducted.
Mil 1 Some practices are performed but not scheduled.
Mil 2 Procedures are documented, and resources provided to support securing the domain.
Mil 3 Policies guide procedures. Clear roles and authority are allocated. Staff have the required skills.
As an example, consider the Asset domain. The NIST framework first factor is the Identity factor. The role of this factor is to document the physical devices, applications, and data types in a business’ digital platform.
Using the Cybersecurity Capability Maturity Model (C2M2), the following MIL levels for the Asset domain are:
- Mil 0 – No activity (unlikely as every business will have made some attempt at an inventory)
- Mil 1 – Important assets are documented in an ad hoc and unplanned manner
- Mil 2 – A partial inventory exists as a device registry with priority of protection given to assets that, in the event of being cyber attacked, present the greatest potential loss to the business. The registry contains relevant information.
- Mil 3 – Includes Mil 2 along with: a complete inventory that is current and updated regularly. When devices, applications and data are no longer needed, they are securely removed and deleted or otherwise destroyed as needed.
Reading
Learn more about the C2M2 model at this link: Cybersecurity Capability Maturity Model (C2M2) | Department of Energy
Evaluating the Effectiveness of Strategies
While frameworks and models (such as those provided by NIST and CIS) provide roadmaps for optimal cyber security environments, they do not provide a mechanism to measure the effectiveness of cyber security strategies – is a business effectively protecting its digital platforms?
Evaluating effectiveness requires a longer view than evaluating individual changes to a business’ cyber protection. To evaluate effectiveness, a business should:
- review data from monitoring over time and measure it against agreed KPI’s (such as Cost per cyber incident). If KPIs are not met, improvements to meet KPIs are implemented as needed, or the business can review the KPIs to ensure they are as needed and realistic.
- examine the cyber security maturity of the business using a cyber security maturity model. The maturity model will assess the business’s current state of protection for a particular cyber security domain. The assessment rating is then compared with that expected by the business.
- perform penetration testing every 6 months and conduct refresher cyber security training every 4-6 months for all staff. This consolidates cyber security awareness and establishes the current level of technical protection.
- monitor of network, application, and system activity to validate effectiveness.
Case Study
Using a Maturity Model to Evaluate Cyber Security
Ace Pty Ltd’s cyber security manager conducts a cyber security maturity assessment using the C2M2 model.
Assessing the Asset domain, the manager finds that the business is rated Mil 1. Despite policy requiring new digital devices and applications be added only after informing Ace’s cyber security team, several new laptops have been found connected to the Ace network that are not in the Ace digital device registry.
Ace’s stated desire is to meet Mil 2 for the Asset domain. Ace’s cyber security manager identifies that the policy on additional technology is not being complied with as required.
To correct this, the manager:
- blocks the unknown devices from the Ace network
- when the device owner’s report issues with connectivity, each owner is required to provide device information for the registry and to advise on the process they used to acquire the device.
For each reported device, the Ace IT department was found to be responsible for the device installation. As a result, the ACE IT manager was advised to ensure that in future any new device was added only after advising Ace’s cyber security team.
Key Points
- A cyber security maturity model addresses a business’ level of implementation of cyber security practices
- Maturity models use different classifications, but all are looking to fully deliver the requirements of an extensive cyber security framework
- Evaluation of strategies can combine KPIs, maturity models, training, monitoring and penetration testing.
Watch
Learn more about metrics and performance in cybersecurity. Watch the video below:
Learn more about cybersecurity maturity. Watch the video below:
Depending on the results of a cyber security strategy change, corrective action may or may not be required. A change is a modification that updates a specific cyber security element, rather than being an assessment of a business’ total cyber security approach.
Corrective action is needed when:
- there has been an oversight (the change requires additional unforeseen changes to be effective)
- there has been a lack of consultation to explore the impact of the change on the business
- monitoring of the strategy post implementation finds that the change is ineffective.
Some examples of changes to policy following evaluation may be that:
- an evaluation found that patches were not applied fast enough to secure the business. This lengthens the window of vulnerability. As result, the strategy is adapted to include vulnerability scans to run weekly rather than every two weeks.
- an evaluation found that employees were continuing to use public Wi-Fi despite policy dictating this is now allowed. A potential change to cyber security strategy could be to turn of public Wi-Fi capability on all company devices.
Case Study
Monitoring to Evaluate Change Effectiveness
Ace Pty Ltd’s management approve a change suggested by their cyber security manager. The change created a new policy and logon procedure for external mobile users. Ace’s remote staff are now prohibited from using public Wi-Fi. Two weeks after the policy was communicated to all staff, a network report shows that a few staff are not using the 4G/5G network provided to their laptops and revert to public Wi-Fi.
After discussions with staff, it became evident that the reason for this is that in some locations the 4G/5G network did not provide a stable connection.
The cyber security manager decided that:
- using public Wi-Fi is high risk
- any inconvenience is momentary, and users can wait to find locations that support 4G/5G.
The cyber security manager takes a suggestion to management to update laptop network software to block Wi-Fi. The suggestion is agreed to by management.
The cyber security manager then:
- arranges for Ace IT to make a network patch and install it to all laptops asap
- sends an email to all staff reinforcing the need to use 4G/5G and the blocking of Wi-Fi.
If an update to a strategy is required, seeking approval to remedy an unexpected consequence of unsuccessful modification is required.
As a rule, whenever change is brought to management for approval:
- first perform a strenuous examination to ensure the change is required and will perform as expected. To do so, testing changes on test systems (for technical changes) is advisable. Further, discussing the potential for a change with affected teams in a business is important to gather support and potentially gather alternative solutions. This includes any technical teams who implement changes on behalf of cyber security managers.
- ensure that management is aware that post change there will be an evaluation to ensure the change is successful. Further, you will advise management if a change is successful or requires additional modification.
To review the effectiveness of cyber security strategies, measuring KPIs before and after a change can be combined with:
- reviewing operational efficiency after the change (did the change alter the businesses workplace as expected or has there been unexpected consequences)
- performing penetration testing to ensure that the change blocks the cyber threat it was intended to negate.
Having to modify either a procedural or technical cyber security change shortly after implementation typically means there has been an oversight. To avoid backing out changes or having to adjust them, consultation with key departments in a business (such as compliance or IT) can often help pick up issues early.
Working across a business with various teams assisting in implementing or working with proposed cyber security improvements brings the opportunity to leverage alternative expertise. Such expertise may provide a perspective that is foreign to cyber security thinking yet solves the cyber security requirement in an ideal way.
Activity 2
