The call came at 2 AM. A major U.S. telecom provider, critical infrastructure supporting millions of cell phone users, was under active ransomware attack. Systems were encrypting rapidly across their network. Within hours, the FBI was coordinating response efforts, executives were in crisis mode, and a specialized team was rebuilding Active Directory from scratch while the clock ticked. For Eric Pittman, VP of Cybersecurity at Teradata, this wasn't a tabletop exercise or theoretical scenario. It was a real-world crisis that revealed critical gaps in how organizations prepare for and respond to ransomware attacks.
The insights from this incident offer valuable lessons for any organization looking to strengthen their ransomware defenses before they face a similar crisis. The lessons learned from those intense days offer a blueprint for building true ransomware resilience.
Inside a Real-World Ransomware Crisis
When the attack hit, the telecom provider faced an immediate cascade of failures that extended far beyond IT systems:
-
Communication breakdown: Primary channels went dark, hampering coordination
-
Decision paralysis: Leadership scrambled between technical recovery and business continuity
-
External pressure: FBI coordination, customer communications, regulatory requirements
-
Time pressure: Every minute offline meant service disruption for millions
"We had to rebuild their entire Active Directory from scratch within hours," Pittman recalls. "They didn't know how the attackers got in or what else was compromised, so they were literally starting from zero."
Although the organization had security measures in place, it lacked the resources needed to navigate through the total business impact, attack surface reduction, and visibility of the compromise.
Lesson 1: Automation is Your Most Effective Ally
It’s easy to create a ransomware incident response plan using templates or open-source guides. The real challenge, and the true mark of readiness, lies in having the resources and automation in place to execute that plan at scale.
The Solution: Leveraging automation is critical. A well-designed playbook becomes an operational blueprint when it is supported by automated tools that can quickly enact its steps.
Modern device management platforms like Kandji are a prime example of this. They operationalize these playbooks by providing centralized policy deployment and rapid response capabilities across all managed Apple devices. Instead of manually pushing out security configurations or isolating devices one by one, automation allows you to do it at fleet scale and at machine speed, drastically reducing the attack surface for ransomware.
Lesson 2: The Power of Immutable Backups
The Game Changer: As Pittman learned from this incident, "the biggest takeaway was immutable backups" because "if they had those, it wouldn't matter if everything else got encrypted. You could just blow that storage away, restore the unencrypted backups and be on your way."
This single insight represents the difference between a manageable incident and a catastrophic event. Traditional backup approaches often fail against modern ransomware because they share authentication mechanisms with primary infrastructure.
Building True Backup Immunity
Technical Requirements:
- Separate authentication systems for backup infrastructure
- Air-gapped or offline copies of critical data
- Regular restoration testing (not just backup verification)
- Documented procedures accessible when primary systems are down
- Retention policies balancing storage costs with recovery needs
The Reality Check: Most organizations discover backup failures during recovery attempts, not during routine testing. Build your backup strategy assuming your primary authentication systems are compromised.
Strategic Integration: Combine immutable backups with automated endpoint protection to create multiple layers of defense. Kandji's security capabilities help prevent initial compromise while ensuring rapid restoration remains viable even in worst-case scenarios.
Lesson 3: Crisis Coordination Extends Beyond Your Organization
The third lesson from the telecom infrastructure attack involves the complex web of external relationships that become critical during a major security incident.
When ransomware strikes, the response extends far beyond your internal team. In the case Pittman described, this included:
- FBI Coordination: Sharing threat intelligence and potentially identify the attackers
- External Legal Counsel: Consulting with legal experts who specialize in cyber law to navigate regulatory requirements and advise on payment implications
- Public Relations and Communication: Managing communications with customers and partners affected by service disruptions
- Technical Specialists: Engaging with pre-identified incident response and recovery experts to assist with remediation
Proactive Relationship Building
These external connections you’ll need can't be established mid-crisis:
- Law enforcement: Establish contacts and understand reporting procedures
- Legal counsel: Pre-select and engage with legal firms specializing in cyber incidents
- Communication partners: Have alternative channels ready for both internal and external communication
- Technical specialists: Vet and retain third-party incident response and forensics experts
Building Resilience Before You Need It
The telecom provider's experience demonstrates that ransomware preparedness isn't just about preventing attacks—it's about building operational resilience that serves your organization every day.
Immediate Action Items
- Audit Your Automation: What automations are in place to reduce your attack surface and accelerate your response?
- Test Your Backups: Can you restore critical systems using only your backup infrastructure?
- Map External Dependencies: Who would you need to coordinate with during a crisis?
Technology as an Enabler
Modern device management solutions provide the automation and controls needed to establish security readiness foundations. Platforms like Kandji enable organizations to:
- Automate critical security configurations across entire device fleets
- Enforce compliance with industry standards and internal policies
- Establish recovery protocols that minimize downtime during incidents
The Time to Prepare is Now
Pittman's experience reinforced a critical truth: "Have multiple plans, multiple playbooks for whether it's ransomware, a malware outbreak, any kind of zero day. What if your primary communication method goes down? Whether that's Zoom or Teams or something else. Have all of these playbooks ready. Be ready for any scenario, not just ransomware." It's now, while you have the luxury of thoughtful planning rather than crisis response.
By learning from real-world experiences and implementing a strategy built on automation, immutable backups, and crisis coordination, organizations can transform potential disasters into manageable incidents.