The EPA's national cyber drill this week exposed a troubling reality: when asked to simulate a complete communications blackout, only 67 out of 390 participating utilities attempted the manual operations module. Just one confirmed they'd actually completed it.
This isn't about lazy operators or underfunded programs. It's about a fundamental misunderstanding of what "contingency planning" means when your SCADA systems can't phone home.
Why These Mistakes Keep Happening
Water utilities operate in a regulatory environment that emphasizes documentation over demonstration. Your incident response plan sits in a binder, reviewed annually, signed off by management. But when Brandon Carter, EPA Senior Cybersecurity Specialist, asked participants to actually take systems offline during the drill, the silence was deafening.
The problem compounds because water utilities vary wildly in size, technology, and complexity. As one sector expert put it: "If you've seen one water utility, then you've seen one water utility." This diversity makes it easy to dismiss exercises as "not applicable to our environment" without testing that assumption.
Mistake 1: Confusing Local Operation with Manual Operation
Why it happens: Your remote pumping stations have local automation. Tank levels trigger pumps without central oversight. When connectivity drops, these systems keep running. You call this "manual operation" in your contingency plan.
The consequence: During the EPA drill, participants described scenarios where treatment decisions had to balance water quality against system pressure. One operator noted: "A decision that's there is, are we going to be trying to fully treat [our water], or are we just going to keep the system full, so we can fight fires and maintain pressure." Local automation can't make that call. If you haven't identified which decisions require human judgment versus which can run on local logic, you'll discover the gap when communications fail.
The fix: Map every remote facility to one of three categories: fully automated with safe failure modes, locally automated requiring periodic checks, or requiring continuous operator presence. For that third category, document the specific manual procedures. Don't write "operator will monitor tank levels." Write: "Operator checks gauge every 30 minutes, manually activates Pump 2 when level drops below 15 feet, documents reading in logbook."
Mistake 2: Planning for Internet Outage, Not Communications Blackout
Why it happens: Your disaster recovery plan addresses "loss of internet connectivity." You've got backup cellular modems. Your incident response runbook assumes SMS-based alerting and mobile email work during an outage.
The consequence: The EPA's scenario simulated a tier-one telecommunications provider suffering a cyberattack by the Chinese threat group Salt Typhoon. No cellular. No landlines. No SMS. No cloud services, email, document sharing, or timesheet apps. Your backup communication plan just evaporated because it assumed partial connectivity, not zero connectivity.
The fix: Test your communication tree with zero electronic connectivity. Can your shift supervisor reach the on-call engineer using only radio or in-person contact? Do you have physical maps showing which staff live within 20 minutes of each facility? One drill participant noted they'd need "24-hour personnel present" at multiple locations with "multiple operators there at the same time." Calculate whether you have enough qualified operators within driving distance to sustain 12-hour shifts for 72 hours. If you don't, that's your real contingency gap.
Mistake 3: Treating the Drill as a Tabletop Exercise
Why it happens: Live exercises disrupt operations. Taking systems offline requires management approval, coordination with multiple departments, and acceptance of risk. A tabletop discussion feels safer and easier to schedule.
The consequence: Of the 67 utilities that selected the manual operations module, only one confirmed completion during the drill. Four more said they'd complete it "after today." Talking through a scenario reveals gaps in your documentation. Actually executing reveals gaps in your capabilities. The Massachusetts control system engineer who noted "most utilities have either private fiber or radio, and so this scenario wouldn't be significantly impactful" made an assumption that needs testing, not assertion.
The fix: Start small. Pick one remote site. Schedule a 4-hour window. Disable its connectivity and require operators to maintain service using only local controls and physical site visits. Document every workaround, every delayed decision, every moment when someone said "normally I'd just check the dashboard." Those moments are your real response requirements. Run this drill quarterly, rotating through different facilities and different shifts.
Mistake 4: Assuming Size Equals Complexity
Why it happens: You serve 120,000 customers across a metropolitan area. Your systems are sophisticated, your SCADA network extensive. A small rural utility serving 500 customers can send "a guy in a truck" to check sites manually, but you can't.
The consequence: This logic leads to paralysis. You conclude that manual operation is impossible at your scale, so you don't plan for it. During the drill, participants serving small communities reported confidence in their ability to operate manually. Larger utilities described tradeoffs between treatment quality and system pressure, suggesting they'd discovered these dilemmas during the exercise rather than before it.
The fix: Decompose your system into operational zones. Which treatment plants, pumping stations, and distribution areas can function independently for 72 hours? For each zone, identify the minimum monitoring and control actions required to maintain safe service. You're not planning to operate your entire system manually. You're planning to keep critical zones functional while communications are restored. That's a solvable problem if you define it correctly.
Mistake 5: Writing Procedures Without Decision Thresholds
Why it happens: Your manual operations procedure says "maintain water quality standards" and "ensure adequate system pressure." These sound like clear objectives.
The consequence: When the drill participant described choosing between full treatment and maintaining pressure, they revealed the gap. Your procedures don't specify the decision threshold. At what point do you accept partially treated water in the system to maintain fire suppression capability? Who makes that call? What regulatory notifications does it trigger? These aren't questions you want operators debating at 4:45 a.m. during an actual incident.
The fix: For every manual operation scenario, document the decision tree. "If tank level drops below X feet and Pump 1 cannot be manually activated within Y minutes, contact Shift Supervisor to authorize Pump 2 activation using untreated water. Notify State Drinking Water Program within Z hours per [specific regulation]." Include the regulatory citation. Include the phone numbers that work without internet. Include the physical location of backup documentation.
Prevention Checklist
Before your next audit or regulatory review:
- Conduct a 4-hour live drill at one remote facility with zero connectivity
- Map all remote sites to automation categories: fully automated, periodic checks required, or continuous presence required
- Calculate operator coverage: can you staff critical sites with 12-hour shifts for 72 hours using only local personnel?
- Document communication tree that works without cellular, landline, or internet (radio frequencies, physical rally points, runner protocols)
- Identify decision thresholds for treatment vs. pressure tradeoffs with specific regulatory notification requirements
- Test backup documentation: can operators find and execute procedures using only paper copies stored at each facility?
- Schedule quarterly rotation through different sites and different shifts
The EPA's drill simulated a "worst-case scenario" involving "sophisticated real-world adversarial tactics." The scenario wasn't hypothetical. It was based on intelligence about Salt Typhoon. Your contingency plan needs to work in that scenario, not just in the one where your backup cellular modem saves the day.
