Lumenly
Back to Blog
Epic outage first stepsEHR outage responseEpic downtime activationhospital cyberattack response first dayEpic downtime proceduresransomware hospital first 24 hoursdowntime communication planincident command EHR outage

The First 24 Hours of an Extended Epic Outage

What the first day of an extended Epic outage demands: activation, paper operations, the two-team structure, and the checklist that sets the trajectory.

Ivan Ip, MDIvan Ip, MD·3 min read·September 13, 2026
The First 24 Hours of an Extended Epic Outage

Hour zero: the activation call

The most expensive minutes of any outage are the first ones — when everyone suspects it's bad and nobody's sure whose call it is. Your activation protocol needs three things, each with a name attached, decided long before the day: a threshold (the conditions that trigger downtime procedures — not "significant disruption," but concrete states), an authority (a role, not a committee, empowered to make the call at 2am without convening anyone), and a channel (how the decision reaches every unit when email and messaging may themselves be down).

Two failure modes dominate real incidents. The first is delay — hoping it's a blip, losing an hour of clean paper documentation because nobody wanted to overreact. The second is fragmented activation — some units on downtime procedures, others still trying to use a dying system, producing a documentation record that's half paper, half electronic, and fully chaotic to reconcile later. A decisive, house-wide call beats a perfectly calibrated late one.

And in a suspected cyber event, one more early decision belongs to hour zero: engaging your security response, which may deliberately sever systems and connections that still appear to work. Painful, correct, and much better executed early.

Hours 1–4: stand up paper operations

Activation is a sentence; operations are the work. Each unit needs its downtime box open and verified — current forms, downtime procedures, the paper MAR process, the phone tree. This is the moment every quarterly cabinet audit either pays off or gets its revenge: forms three build-versions old, a downtime workstation that's been unplugged for months, a phone tree naming people who left.

Three workflows deserve immediate leadership attention because they're where downtime hurts patients fastest: medication administration (no scanning, no interaction checks — pharmacy needs a visible, forward-deployed presence), results routing (labs and imaging keep running; the results need a physical path back to the ordering clinician, with an escalation route for criticals), and the paper census (from hour one, every admission, discharge, transfer, and death captured by unit, consolidated centrally each shift — this document becomes the foundation of your entire recovery, and it only exists if someone starts it now).

Hours 4–12: split into two teams

Here's the structural move most organizations miss, and it may be the most important paragraph in this post: recovery planning starts on day one — not when the system comes back.

Stand up two teams with distinct mandates. Team one runs the downtime: unit support, workarounds, safety surveillance, communications. Team two ignores today's fires entirely and starts planning the comeback: cutover criteria, the batch job review with your EHR vendor's technical team, backfill scoping, staffing math for the Tier 2 marathon. If everyone fights today's fire, the recovery gets planned in week two by exhausted people — which means you buy days of extra chaos at exactly the moment the organization can least absorb them.

This is also the window to formally engage your EHR vendor's technical team and open the recovery conversation — not at restoration, now.

Hours 12–24: establish the cadence

By the back half of day one, heroics need to become rhythm. Set the communication cadence and honor it ruthlessly: updates to staff at fixed times, whether or not there's news. "Here's what we know, here's what we don't, here's when you'll hear from me next" — delivered on schedule — builds more trust than confident guesses that age badly. Silence gets filled by rumor within hours.

Stand up the daily incident huddle: same time, same place, status pre-read, the meeting itself reserved for decisions with the decision-maker in the room. Start the decision log — what was decided, by whom, when, knowing what — because six weeks from now, "why did we do X?" will matter to the retrospective, the auditors, and possibly the lawyers, and nobody will remember.

And before midnight, plan the thing day one always forgets: relief. The people who carried hour zero cannot also carry day three. Name deputies and force rest rotations now, while adrenaline still makes everyone feel inexhaustible. Extended outages are marathons that always begin disguised as sprints.

The day-one checklist

Print this into your downtime plan:

  1. Downtime activated house-wide, by the named authority, via the offline channel.

  2. Security response engaged (if cyber).

  3. Unit downtime boxes open and verified.

  4. Pharmacy forward-deployed; paper MAR process live.

  5. Results routing path established, criticals escalation named.

  6. Paper census started, consolidation cadence set.

  7. Two-team structure stood up; recovery planning underway.

  8. EHR vendor technical team engaged; vendor contacted.

  9. Communication cadence announced and kept.

  10. Daily huddle scheduled; decision log started.

  11. Relief rotations named.

Eleven lines. The honest question — the one worth a tabletop this quarter — is whether your organization could execute all eleven from the plan, or whether day one would run on institutional memory and adrenaline. Memory doesn't scale, and adrenaline runs out around hour 30. The checklist doesn't.