My Experience with ICS Security and Compliance in an Electric Utility Environment
After spending some time reflecting on the recent SANS ICS Security Summit with virtual meetings around ICS and IT/OT security, I’ve come to the conclusion that the industry has made significant progress and very good ideas and valuable techniques were discussed. However, the frequency and severity of cyber threats keep accelerating and we still have a lot to do before reaching the level of resiliency needed to protect our environments.
Here are my insights, after 15 years working directly in the utility environment, into the challenges we face and how to overcome them. I believe the ability to fully assess environments and apply the security and compliance standards and controls to ICS systems is extremely difficult because of: 1) legacy systems, 2) lack of resources, and 3) insufficient training. Below is a hypothetic scenario to illustrate those challenges.
Scenario:
A utility has 5 states of coverage and it took several mergers and acquisitions over 40 years to achieve the coverage they currently have.
- Environment and culture are not standard – I have visited many utility substations and plants doing security and compliance assessment and the utility environment and culture can be very different from State to State or even among business units. The operational team from one site may not have the ability to support and maintain legacy equipment or have the knowledge of which systems are running on a different site.
- Visibility remains a critical challenge – Mandatory regulations that cover the OT assets can assist cyber security team in gaining the visibility but only to a certain extent. The North American Electric Reliability Corporation (NERC) has a number of requirements that govern the utility environment, and the United States Nuclear Regulatory Commission (USNRC) has implemented the 2013 Cyber Security Directorate to centralize oversight and protect digital computers, communications systems and networks. Elements of a compliance program that are key to document the environment and verify security controls include:
- Asset management: this function is a must for any assessment but is difficult to maintain up-to-date.
- Identity and access management: this is also a must-have but often take years to integrate into an OT substation environment since it is expensive and the large number of substations and plants with dissimilar systems creates integration issues.
- Risk assessments, vulnerability management, and change management: these functions involve identifying threats, vulnerabilities, and managing patches. The core challenge is that vulnerability scanners can rarely be launched in a substation or plant environment due to the risk of breaking equipment. Also, the presence of legacy systems is a major roadblock when implementing a change management program.
- Security controls: these functions include network segmentation and monitoring, which is often made difficult by the lack of logging capabilities.
- Physical security: this is one of the primary security control and is often a challenge due to the large number of physical keys and people who have to access them.
Solution:
Addressing those challenges requires investments and a multi-faceted approach. In particular, I would recommend:
- Understanding how vast the environment is and spending the time to capture the specificities of each site and system
- Developing a strong training program to address the short supply of OT engineers
- Leveraging regulations as a forcing function to align different teams behind the same objective. It is also important to know that NERC CIP allows for Technical Feasibility Exceptions (TFE) to address the limitations of legacy equipment.
- Building incentive for business owners to upgrade their environment
- Expanding system and network visibility through logging and alerting in order to eliminate blind spots and to develop situational awareness
In summary, we have a long road to travel to make our ICS environments resilient to cyber attacks, but we can start by addressing the following challenges:
- Hiring the right people with the right skills
- Leveraging independent monitoring technology that does not impact the operation of the systems during an assessment
- Progressively replacing legacy systems and requiring vendors to have their base systems assessed prior to implementation and turnover
