When more than 30 Minnesota community water systems lost automated control over two days this week, utility crews had to manually operate pumps, valves, and treatment processes while state and federal teams scrambled to contain the damage. The questions that followed are the same ones your team will face if operational technology gets compromised.
These questions highlight that OT incidents can reveal gaps in coordination, authority, and technical capability that your tabletop exercises might have glossed over. Here's what practitioners actually need to know.
Can we actually fall back to manual operations if controllers fail?
You need documented manual procedures for every automated process, and operators who've practiced them recently.
When Braham, Minnesota's well and water treatment plant went offline, public works crews restored operations in under two hours because they had contingency procedures ready. Plymouth maintained normal operations at two water towers and multiple wastewater lift stations using manual controls overnight.
This isn't about having a binder on a shelf. Your operators need to know which physical valves correspond to which SCADA points, where the manual overrides are located, and what safe operating parameters look like without automated feedback. If your team hasn't run a manual operations drill in the past six months, you're relying on institutional memory that might not exist when you need it.
Document the sequence for each critical process: what to check first, what can wait, what combinations are unsafe. Include physical locations, valve numbers, and contact information for specialists who know the equipment. Test these procedures during planned maintenance windows, not during an incident.
Who coordinates when state, federal, and local teams all show up?
Establish clear authority and communication channels before an incident, because you won't have time to negotiate roles while systems are down.
Minnesota activated its Whole-of-State Cybersecurity Program, which brought together MNIT, the Department of Public Safety, the Bureau of Criminal Apprehension's Minnesota Fusion Center, state health and pollution control agencies, CISA, the EPA, and the FBI. That's at least seven organizations that need to share information, avoid duplicating work, and not contradict each other when talking to the public.
Your incident response plan should specify who has authority to make operational decisions (usually you), who coordinates technical investigation (often a state fusion center or sector-specific agency), and who manages public communication (typically your communications team with input from state officials). Create a single shared channel for real-time coordination and a daily sync meeting for strategic decisions.
Define what information you'll share with each partner and through what mechanism. CISA needs technical indicators of compromise. Your state environmental agency needs water quality data. The FBI needs forensic evidence preserved properly. Set these expectations in your plan, not during the crisis.
What do we share with other utilities, and how fast?
Share indicators of compromise and attack patterns immediately through established sector channels, even if attribution is uncertain.
MNIT's cybersecurity teams shared threat intelligence and indicators of compromise with affected utilities as the investigation progressed. This let other water systems check their own environments for similar activity before attackers moved laterally.
Your sector likely has an Information Sharing and Analysis Center or similar body. Make sure you're a member and that your security team knows how to submit and receive threat intelligence through it. When you detect suspicious activity, share the technical details: IP addresses, file hashes, unusual network patterns, configuration changes.
Don't wait for perfect information. If you see controllers behaving strangely or unauthorized project file access, notify your sector partners immediately. They can check their logs for similar patterns and implement additional monitoring before an attack succeeds.
The Minnesota incident occurred days after federal agencies updated a joint advisory warning that Iranian-affiliated actors were targeting internet-connected industrial controllers across U.S. critical infrastructure. That timing suggests the threat intelligence sharing system worked, but only for organizations actively monitoring those channels.
How do we know if what operators see on screens is real?
Implement independent verification for critical readings, because attackers specifically target the displays that operators trust.
The federal advisory tied to this incident describes attackers modifying data shown on human machine interface and SCADA displays, disabling critical shutdown and alarm logic so systems drifted into unsafe states while operators saw normal readings. This is the nightmare scenario: your team thinks everything's fine because the screens say so, but physical processes are failing.
You need out-of-band verification for critical parameters. Install independent sensors that report through different network paths. Require operators to physically verify critical readings during shift changes. Set up alerts that trigger if SCADA values stop changing or change too smoothly (which might indicate replayed data rather than live readings).
Train your operators to recognize subtle signs that displays might be compromised: timestamps that don't advance, values that hit exact targets too consistently, or alarms that don't correlate with other system behaviors. Trust operator instinct when something feels wrong, even if screens show normal.
Should we take controllers off the internet completely?
Yes, and route any necessary remote access through authenticated VPN or jump hosts with strict access controls.
Federal agencies warned that Iranian-affiliated threat actors have been exploiting internet-connected operational technology devices, including programmable logic controllers, since at least March. The agencies urged critical infrastructure owners to eliminate direct internet exposure for controllers and other OT devices.
Audit your network to find every device with a public IP address. Programmable logic controllers, human machine interfaces, remote terminal units, and even building management systems should sit behind firewalls with no inbound rules from the internet. If vendors or integrators need remote access for maintenance, require them to connect through your VPN with multi-factor authentication and log every session.
Segment your OT network from IT using firewalls that default-deny all traffic and only allow specific, documented flows. Monitor east-west traffic between OT segments to detect lateral movement. Consider deploying a unidirectional gateway for data flowing from OT to IT, which physically prevents commands from traveling back.
What legal protections apply when we share threat intelligence?
This is currently unsettled, which creates risk for organizations sharing detailed technical information.
Minnesota's Whole-of-State Cybersecurity Program relied on federal liability protections that lapsed last fall and remain unsettled in Congress. These protections were designed to shield organizations from lawsuits when they share cybersecurity threat information with government partners.
Consult your legal team before establishing threat intelligence sharing relationships. Understand what protections exist under current law, what information you can share without exposing proprietary details or customer data, and how to document that sharing occurred in good faith for cybersecurity purposes.
Even without perfect legal clarity, sharing indicators of compromise through sector ISACs generally carries lower risk than sharing detailed vulnerability information or incident response strategies. Focus on technical indicators (IP addresses, file hashes, attack patterns) rather than organizational details about how the attack succeeded.
How do we test incident response without disrupting service?
Run tabletop exercises that simulate the coordination challenges, not just the technical response.
Most incident response exercises focus on technical containment: isolating compromised systems, preserving forensic evidence, restoring from backups. But the Minnesota incident shows that coordination across multiple agencies and maintaining public confidence are equally critical.
Design exercises that force your team to make decisions with incomplete information: Should you shut down systems preventively if you detect reconnaissance activity? When do you notify the public? How do you coordinate with state agencies that have regulatory authority over your operations? What happens if manual operations extend beyond one shift?
Invite external partners to participate: your state fusion center, sector-specific regulators, local emergency management, and key vendors. Test the communication channels and decision-making authorities you've documented. Identify gaps in your plans while you still have time to fix them.
Where do we go from here?
Start with the joint federal advisory on OT device exploitation. It includes specific technical mitigations and detection methods based on observed attacker behavior.
Review NIST Cybersecurity Framework 2.0 functions for Detect, Respond, and Recover, mapping your current capabilities against each category. Use ANSI/ISA-62443 to guide OT network segmentation and access control decisions.
Connect with your sector's Information Sharing and Analysis Center if you haven't already. The value of threat intelligence increases exponentially when you're part of the community receiving and contributing to it.
And test your manual operations procedures this quarter. When automated controls fail, your operators' muscle memory and documented processes are the only things standing between disruption and disaster.
