The Conventional Wisdom
When security leaders discuss operational resilience for critical infrastructure, manual operations often emerge as the ultimate fallback. The logic seems sound: if hackers disable your SCADA systems, cloud services, and internet connectivity, you send operators to each site with clipboards and radios. Problem solved.
The EPA recently conducted a national cyber drill that reinforced this thinking. The scenario simulated a complete communications blackout at water utilities, forcing them to operate without internet, cellular, or SMS. The agency positioned manual operations as the answer to worst-case cyberattacks.
Here's the problem: for most water utilities, true manual operations aren't a realistic contingency plan. It's theater.
The Reality Check
The EPA drill attracted 390 participants, but only 67 opted for the manual operations module rather than the tabletop exercise. Of those 67, only one confirmed they'd actually attempted a live transition to manual operations during the drill. Four more said they'd do it later.
This isn't apathy. It's a realistic assessment of what's required for manual operations.
Andy Krapf, director of cybersecurity for Loudoun Water, serving over 325,000 customers in Northern Virginia, highlights the distinction: local operation versus true manual operation. When connectivity to a remote pumping station is lost, local automation should continue. Tank levels rise and fall, pumps turn on and off based on demand. An operator checks in periodically, but the system runs itself.
True manual operation means the automation has failed. You need an operator physically present 24/7, watching instruments, manually turning pumps on and off. For a utility serving hundreds of thousands of customers across multiple facilities, that's not a contingency plan. It's a different organization entirely.
Consider what's needed: enough trained operators to staff every critical site around the clock, physical access to sites designed for minimal human presence, manual procedures your team knows how to execute, backup instruments that work without network connectivity, and a command structure that functions without digital coordination tools.
One drill participant noted the immediate staffing challenge: "Locations would need 24-hour personnel present. For some locations, multiple operators would need to be there at the same time." Another said they'd switched from 8 to 12-hour shifts. That's not sustainable for days, let alone weeks.
The Evidence
The drill scenario was based on intelligence about Salt Typhoon, a Chinese cyberthreat group. Carter, the EPA senior cybersecurity specialist leading the drill, said "unfortunately, that part is not made up." The threat is real.
But the responses during the drill revealed what utilities actually face. Several participants acknowledged they'd have to make immediate tradeoffs between water quality and water pressure. One said: "A decision that's there is, are we going to be trying to fully treat [our water], or are we just going to keep the system full, so we can fight fires and maintain pressure."
That's not a fallback plan. That's triage.
A control system engineer from Massachusetts noted that most utilities he knew had non-internet communications with remote sites through private fiber or radio. "This scenario wouldn't be significantly impactful," he said. A smaller utility participant said they'd "go directly to manual operation and notify the town manager right away" because their system was "small and easy for us to manage."
The pattern is clear: utilities that could realistically operate manually are small enough that they barely need automation. Utilities complex enough to need sophisticated automation can't realistically fall back to manual operations for any meaningful duration.
What to Do Instead
Your resilience strategy shouldn't center on manual operations. It should focus on maintaining automated operations under degraded conditions.
First, design your control systems for loss of connectivity. Your remote sites should operate safely in local mode when they can't communicate with the central system. This means local controllers with enough logic to handle normal operations, fail-safe modes when something goes wrong, and instrumentation that works without network dependencies.
Second, build redundant communications paths that don't share failure modes. If your primary connectivity is internet-based, your backup shouldn't rely on the same ISP or physical infrastructure. Private fiber, radio networks, and satellite links each have different threat profiles. One drill participant's comment about non-internet communications shows this works.
Third, define what "degraded operations" means for your utility. You're not choosing between normal operations and manual operations. You're choosing between normal operations, reduced operations with longer response times, operations with reduced treatment capacity, and emergency operations focused solely on maintaining pressure for firefighting. Document the decision criteria and authority for each level.
Fourth, test your local automation regularly. Don't wait for a drill to discover that your remote site can't operate independently because someone centralized a critical function two years ago. Disconnect sites from the network during maintenance windows and verify they continue operating safely.
Fifth, stop pretending you'll staff every site manually. Instead, prioritize which sites actually need human presence during an extended outage and what those operators need to do. One operator checking tank levels twice per shift is realistic. Continuous manual operation of a treatment plant is not.
When Manual Operations Make Sense
Manual operations work as a contingency for small utilities with simple systems. If you serve a few thousand customers from a handful of sites, sending operators with radios and procedures might be viable for days or weeks.
Manual operations also work for specific, time-limited scenarios at larger utilities. If you need to bypass a failed controller or operate a single site while you repair automation, having documented manual procedures and trained operators makes sense.
And the discipline of planning for manual operations forces you to understand your systems. Walking through how you'd operate manually reveals dependencies, single points of failure, and automation you've built that you don't actually understand.
But for most water utilities serving tens or hundreds of thousands of customers, manual operations aren't your resilience strategy. It's what you do when your resilience strategy has already failed. Build systems that keep working when the internet doesn't, and you won't need to find out whether your team can actually run a treatment plant with clipboards.
