This morning’s news started with the report of a ransomware attack on the country’s second largest school system in Los Angeles.
The warnings or reports of cyber security threats occur daily. However, until reading the article below I had not seen an actual account of responding when this happens in a credit union.
The case study appeared in CUSO Magazine. It was written by Matt Sawtell, VP of Managed Technology Sales at CU*Answers who had first-hand experience with the event.
Facts are given and lessons learned. I would urge anyone in this area of responsibility to read this account.
The Anatomy of a Ransomware Incident (And What We Learned)
Following a trend that has been developing over the last ten years, cybersecurity is a topic that is no longer reserved for the dimly lit, garden-level, IT-dwelling teams to consider. It is a topic that is on the minds of those in the boardroom.
As events have garnered ever more concerning headlines, from the Colonial Pipeline incident, which was settled for around $5M in Bitcoin, to the various Microsoft incidents, to the cypto.com hack which saw thieves lift approximately $33M from over 500 user wallets back in January, it’s hard to imagine that no credit union has been affected by an incident in the last few years.
The target that financial institutions have on them is especially large. The attackers believe FIs have the dollars to pay and they possess sensitive member information, which has its own value and adds leverage to a potential payout.
In the last year, we have had the experience of participating in the response to one of these attacks on a credit union. The experience reaffirmed the importance of solid cybersecurity plans and operations as essential, and we gathered some takeaways for others as we worked through the event.
The event unfolds
Friday afternoon, nearly the close of business, our support team received a call from a credit union asking some questions about why their access to data on the network wasn’t working. We followed our normal troubleshooting and escalation protocols. Shortly after digging into this troubleshooting, the bad actor reached out to the credit union to say they had exfiltrated member information out of the credit union and communicated the ransom. Our team escalated this to management, who then advised the credit union to shut down systems and to contact their cybersecurity insurer to begin assisting with the incident.
Cybersecurity insurance is crucial
You have cybersecurity insurance, right? Believe it or not, we have run into organizations recently that do not. We view this as very important coverage, not just because of the financial aspects but also the incident response and forensic and legal resources these insurers can bring to bear in order to minimize the impact of an incident like this.
In this instance the response was swift—a forensic team and case manager were assigned from a firm that specializes in that work. They would quarterback the incident from here through the end. The lead had extensive experience working for a federal agency responding to just these kinds of incidents.
Be diligent, there is a pattern of timing with these events. If you look at the recent rash of events, it seems like the news often breaks on a Friday afternoon, weekend overnight, or before a major holiday. The bad guys know they may have a better go of it when we may have relaxed our guard a bit.
The mitigation work begins
From the initial contact Friday, the case manager was working with multiple parties and coordinating that work on daily (sometimes multiple) calls with all involved. The groups included the forensic team, legal team, negotiator, cybersecurity firm, our CUSO, and the credit union.
We were tasked with a few things at the start, such as determining if we had good backup copies offsite and getting the credit union an alternative way to do some of the daily processing and member work that was needed while the network was shut down. Thankfully, the online and mobile banking systems and audio response were unaffected by this outage so members could still do many of the transactions they needed.
The forensic team was digging in and looking for indicators of compromise (IOCs) as well as any information that might point to a known group of bad actors that pulled off the attack. They used tools, requested hard drives be pulled out of equipment and sent for inspection, and on a daily basis made progress in unraveling the who, when, and how details.
The negotiator was busy interacting with the bad actor and working to negotiate down the initial $5M ransom request. This if nothing else would buy time to decide what the options were over the coming days. The updates from this individual made the whole event seem like a spy movie as much as a cyber incident.
The cybersecurity company utilized tools to start monitoring behavior on the network, process and traffic analysis, and ingress and egress.
The credit union had closed itself to the membership for over a week following the start that Friday. They were present for every call and update, and ultimately made decisions on how all parties would proceed with the work they were doing. In the meantime, they also needed to figure out how to communicate with their members, regulators, law enforcement, and other stakeholders.
Do not overlook a communication strategy
Communication is key. When you have an incident, it is a stressful time. We have all witnessed companies that do a good job of communication and manage the incident well and we have also seen those who… leave room for improvement.
Take for instance the Colonial Pipeline incident. If you lived in the DC/Maryland/Virginia area during this incident, you witnessed panic fuel buying almost overnight. Communication between the pipeline company and the government was not forthcoming where it could have been to calm and inform the public.
On the other side of that are incidents like the Kaseya zero-day from 2021, where the CEO was out in front with regular updates, clients were informed and given IOCs before they were in the news, and the credit union provided transparency and clarity about what to do next.
As a financial institution, one thing to consider is having your incident response strategy and even sample messaging ready to go in advance. Have it cleared through your legal team, management team, and board; keep them in the know on the details and approach. Most importantly, understand that in some cases sharing too little, too much, or speculating publicly can do more harm than good.
You can take this a step further even by conducting tabletop exercises where your team will role-play out various scenarios in order to prepare. Finding someone with experience is a great way to guide the conversation and get the most out of one of these exercises.
Backups and the ensuing recovery
The forensic team started to make headway with their analysis. IOCs were found and pointed back to a foreign group that specialized in gaining access to business networks in the west. They could not tell when that original compromise had happened, but the method they used was sophisticated and had been found at other companies that had similar intrusions.
One of the most interesting things they found was that the state-sponsored group had likely sold access to the credit union network to another, likely an organized crime group that specialized in ransomware. This approach we are told is more common these days as the groups then specialize in their respective areas.
In the meantime, we had validated that offsite backups were not contaminated and could be used to help rebuild the credit union network. The cybersecurity firm had a standard process to create a new, separate, air-gapped network and slowly move machines from the dirty network to the clean one after they had been sanitized. This was painstaking work and took many days to complete. We worked very closely with them and at their direction to ensure the details were followed for each system.
While this was happening, the negotiator continued to haggle with the bad actors over the dollar amount requested. The bad actors had also given proof that they were able to exfiltrate member information, including an AIRES file, from the credit union and were prepared to sell that information on the dark web if their demands were not met. This is a newer tactic to add leverage in the hope of getting a payout.
The credit union was closed for this week while all tech was sanitized. Given they had good backups, and there was no guarantee the bad actors would return the exfiltrated member data, they decided not to pay the ransom, which at this point had been negotiated down to approximately $2.5M. The members who wanted to do in-branch transactions were starting to get frustrated, so re-opening as soon as was safely possible was the highest priority.
A more common and costly occurrence
Ransomware events of several years ago were not nearly as sophisticated or as costly as they had become. Ransomware events were often slow to propagate the network, easy to detect if they had already been in the wild by traditional endpoint security software and the ransoms were in the tens of thousands, not millions of dollars.
The involvement of both states sponsored and organized criminal groups point to how effective a revenue generator this has become for groups that are sheltered in countries that either directly support or turn a blind eye to their activities. Think about the number of companies you have read about that publicly disclose this because they must…then think about the many multiples more that do not.
The end of an incident
Thankfully, this incident had as good an outcome as could be expected. The following week the credit union reopened its doors to members. The credit union retained most of their members, for whom they were providing credit monitoring for the next year. They opted to retain the cybersecurity firm to supplement their efforts after the incident concluded. The remediation and recovery was an effort that required over 1,000 hours of work from multiple teams. Our Network Security team had over 400 hours in the recovery alone.
What are the key takeaways for your institution?
- Have a solid plan. It is best to prepare in advance and avoid trying to come up with a response in the stress of the moment. Prepare communications, understand and have contact information for the key players on your incident response team, and make sure everyone knows their roles and responsibilities.
- Understand the technology and security you have. It is difficult to assess cybersecurity risks and gaps if you do not understand both what your team is doing, what your third parties and partners are doing, and what they are not doing. Make sure you have gone through a detailed assessment of this and that you are comfortable with the residual risk based on your approach.
- Align with the right partners. Consider seeking out specialized partners for things like 24×7 monitoring through a security operations center or a managed detection and response service. Make sure your insurance coverage is appropriate for your organization.
- Test, prepare, and practice. People are key in cybersecurity effectiveness and incident response. Make sure you’re training your team on the threats out there, how to use tech safely within the organization and to report suspected incidents as soon as possible. Conduct tabletop incident response scenarios to practice what an event might look like with your team.