Hacking Operational Technology for Defense: Lessons Learned From OT Red
Teaming Smart Meter Control Infrastructure

High-profile security incidents in the past decade have brought
increased scrutiny to cyber security for operational technology (OT).
However, there is a continued perception across critical
infrastructure organizations that OT networks are isolated from public
networks—such as the Internet. In Mandiant’s experience, the concept
of an ‘air gap’ separating OT assets from external networks rarely
holds true in practice.

In 2018, we released a blog post presenting the tools and techniques
that TEMP.Veles
used during the TRITON incident
to traverse from an external
compromise of the information technology (IT) network of a critical
infrastructure organization to the safety systems located deep in the
OT network. We regularly reproduce this approach in our OT-focused red
team engagements to expose similar attack paths across client
infrastructure and to identify environment specific opportunities to
prevent and detect network propagation techniques across intermediary systems.

In this blog post, we share another case study from one of our OT
Red Team
engagements to illustrate the tactics, techniques, and
procedures (TTPs) that can be leveraged by sophisticated threat actors
to breach the protected perimeter between an IT network and an OT
network. We also examine some of the different types of critical
information often found in IT networks that an attacker can leverage
during later stages of the Targeted Attack Lifecycle. The goal of this
engagement was to access an endpoint meter control infrastructure for
a state-wide smart grid environment from the Internet and turn it off.   

To hear our experts relay more on this and other OT Red Team lessons
learned, join our FireEye
Mandiant Virtual Summit session

Visit our website
to learn more about Mandiant’s OT security practice or contact us
directly to request Mandiant
 or threat intelligence.

Building the Foundation: Information Gathering for IT-OT Network Propagation

Targeted OT attacks attempting to cause physical impacts require
planning. A sophisticated actor who is motivated to disrupt or modify
an industrial process from a public network will necessarily need to
maintain access to the victim environment and remain undetected for
enough time to accomplish their objective. Throughout this time, the
actor will strive to learn about the control process to formulate the
attack, figure out how to pivot to the OT systems and bypass security
controls, and sometimes even develop or deploy custom OT malware.

Similar to the techniques used by TEMP.Veles to reach the OT network
during the TRITON
, Mandiant’s experience during red team engagements
highlights that collecting information from IT network assets plays a
crucial role in targeted OT attacks. As a result, the internal
reconnaissance phase for OT targeted attacks begins in the enterprise
network, where the actor obtains knowledge and resources to propagate
from an initial compromise in the IT network to remote access in the
OT network. Detailed information collected about the target, their
security operations, and their environment can also support an actor’s
attempts at remaining undetected while expanding operations.

Hacking Operational Technology for Defense: Lessons Learned From OT Red
Teaming Smart Meter Control Infrastructure

Figure 1: Targeted OT attack from a
public network

Thinking Like an Adversary: How to Turn Off Smart Energy Meters

The ideal scenario for an attacker targeting OT systems is to
achieve their objective while remaining undetected. Mandiant’s Red
Team works with clients across critical infrastructure industries to
simulate attack scenarios in which actors can accomplish this goal by
gaining access to OT systems via compromise of external facing IT
networks. During these engagements, we emulate real actor behaviors to
learn about our target and to determine the best paths for IT/OT
network propagation.

For this engagement, we simulated an end-to-end OT-specific attack
scenario in which we tested the security controls and response
capabilities of an organization to protect smart grid meter control
infrastructure from an external attacker. Mandiant leveraged
weaknesses in people, process, and technology to gain remote access
from the public Internet and to achieve a set of pre-approved
objectives in the OT environment.

Establishing a Foothold in the IT Network

Mandiant conducted a spear phishing exercise to gain initial access
into the client’s enterprise network from the Internet. We defined a
combination of two different phishing scenarios that we deployed to
test email filtering and egress monitoring controls:

  • Embedded link for a malicious file hosted on a Mandiant owned
    domain on the Internet
  • Email attachment for a Microsoft
    Office document with auto – executable macro code

This exercise allowed our team to achieve code execution on a user
workstation in the enterprise environment and to establish an
unattributable egress communication path to a Mandiant hosted Cobalt
Strike Command and Control (C&C) server on the Internet. After
establishing a stable communication path with workstations in the
enterprise environment, we utilized the following publicly available
offensive security tools (OST) to escalate privileges and to obtain
domain administrator level access:

  • ldapsearch to
    enumerate information in the enterprise domain
  • PowerSploit
    to exploit common security misconfigurations in IT
  • WMImplant
    to move laterally from one system to another in the internal
  • Mimikatz
    to extract credentials for local user and domain administrator

As domain administrators, we gained unrestricted access to a variety
of resources connected to the enterprise domain (e.g. server
resources, file shares, IT applications, and administrator consoles
for IT systems). During the initial stages of our engagement, our
actions were in no way different to other less sophisticated
intrusions on industrial organizations, such as financially-motivated compromises.

Defining Our Path to the OT Network

Similar to real world threat actors carrying out targeted OT
attacks, Mandiant’s OT Red Team dedicates significant effort for
internal reconnaissance in the IT network to develop a logical mapping
of the extended network architecture and discover targets of interest
(people, processes, or technology). The information we acquire helps
us to (a) define paths to propagate from the IT to the OT network and
(b) achieve our final objective in the OT network without raising
alarms. During OT Red Team engagements across different industries, we
follow a real attacker’s cost-benefit analysis to determine which
sources or methods are most likely to help us obtain that information.

Figure 2: Information sources and target
information from enterprise networks

For this engagement, we leveraged the domain administrator
credentials obtained in the previous phase to gain access to Microsoft
System Center Configuration Manager (SCCM) in the IT network. Logged
into the SCCM console, we leveraged software deployment features for
collection to establish C&C over user workstations belonging to
pre-selected departments in the client organization.

Mandiant chose the specific groups based on the names of their
departments and the description attributes, which suggested a high
likelihood of member users with high privilege access for network
infrastructure or application management. This included members of the
following groups: network management, firewall administration, control
engineering, and smart meter operations.

Access to user workstations of target employees in these departments
enabled us to:

  • Capture keystrokes to obtain remote desktop protocol (RDP)
    credentials for the OT network by using a Cobalt Strike modified
  • Login to department file shares and extract OT system
    design documents
  • Extract network design documents and
    backup files for OT firewall configurations found in the firewall
    management console
  • Find plaintext credentials for OT
    management systems from operation manuals

Internal reconnaissance in the IT network not only allowed us to
obtain remote access credentials for the OT network, but to also gain
a deeper understanding of the business processes and technical control
system operations in the OT environment by reviewing internal
OT-specific operational procedures and security documentation such as
asset inventories and configurations.

Propagating to the OT Network

During the process of propagation from IT to OT networks, an actor
will leverage previously compromised systems, credentials, or
applications to access systems in higher security zones—such as OT
demilitarized zones (DMZ). Based on our observations during multiple
red teaming engagements and research, the most likely attack vectors
for propagation are:

Table 1: Most likely attack vectors for
IT/OT propagation

For this engagement, we initially analyzed the system architecture
to define the best path to follow. Engineers from the target
organization were required to use multi-factor-authentication (MFA) to
gain remote access to jumpbox servers in the OT network. While not
impossible, bypassing this setup would require more time and
resources. We instead decided to search for other plausible attack
propagation paths.

Figure 3: Formal communication path from
enterprise to OT network

Reviewing the firewall configuration files, we identified a
dedicated communication path for management access to a Microsoft
Windows patch management server in a DMZ between the IT network and
the OT network. This patch management server was running on a virtual
machine in the DMZ network, while the administration console for the
underlying hypervisor software itself was hosted in the IT network.

Mandiant logged into the administration console for the hypervisor
software using IT network domain administrator credentials. We then
leveraged guest machine administration features via direct console
access to execute commands on the patch management server in the DMZ
network. The compromise of the patch management server in the DMZ
allowed us to pivot via SMB connections to Microsoft Windows-based
intermediary systems in the OT network.

Figure 4: Remote attack propagation path
from IT network to OT network

Lastly, we compromised Microsoft Windows server systems in the OT
network to complete the objectives of the exercise. Using OT
credentials retrieved in the previous phases, we authenticated to the
SMB service (using single factor authentication) by pivoting through
the patch management server in the DMZ network. This enabled us to
execute remote console commands on management servers (such as the
domain controller) in the OT network.

With access to the domain controller in the core OT network, we
extracted credentials for high privilege domain administrator accounts
in the OT network using DCSYNC and Mimikatz. Using these accounts, we
gained control of management servers, application servers, and
operator workstations in the OT network. Mandiant was also able to use
compromised credentials to login to the human machine interface (HMI)
portal for the meter control infrastructure and issue a disconnect
command for a target endpoint meter in the smart grid environment.

Strategic Collection and Detection Opportunities During
Reconnaissance and Network Propagation

Although specific capabilities such as malware and tooling vary
amongst incidents, internal reconnaissance and network propagation are
consistently needed for sophisticated adversaries to expand remote
operations from external networks to OT systems. Focusing collection,
detection, and hunting efforts on assets or information that are
likely to be compromised during these phases presents defenders with
strategic opportunities to hunt for and detect targeted adversary
activity before it poses a risk to control systems.                   

  • In a previous blog post stating our approach to OT security,
    we highlighted that IT networks close to production networks and OT
    intermediary systems remain the best zones to detect OT targeted
    attacks, a.k.a. “The
    Funnel of Opportunity
    ”. As actors pivot across systems and
    networks to gather information and elevate privileges, they leave
    footprints that can be tracked before they propagate to critical
  • An actor who covertly performs internal
    reconnaissance and propagates to the OT network is already
    positioned to cause damage on mission critical assets and is
    unlikely to be discovered. Early detection of adversary activity
    before reaching critical OT systems will decrease the dwell time and
    the risk of an incident.
  • OT defenders can prioritize
    collection and detection, alert triage, and incident response
    efforts by becoming familiar with the types of information and
    services that OT focused threat actors commonly search for during
    internal reconnaissance in IT networks and network propagation
    across OT intermediary systems.
  • Understanding where this
    information resides presents defenders with a catalog of systems and
    networks to focus collection and detection efforts on. Defenders can
    create tailored detections to hunt for adversary activity pursuing
    this information, prioritize alert response efforts, and identify
    additional security controls to implement. Mandiant red teaming in
    OT can help organizations identify which data is valuable for
    attackers to support their network propagation efforts and which
    systems are most likely to be compromised by attackers targeting OT

Visit our website for more
 or to request Mandiant
 or threat intelligence.

By admin