What to Do After You’ve Been Hacked
Protocol hacks are DeFi staples now, with more than $400m stolen over the first 5 months of 2021. Some projects manage to recover from the hacks, while others fall apart.
What determines which projects survive and which ones break under the pressure? There’s no bulletproof formula, but our experience suggests that lack of both preparation and war room expertise turns a hack into a major crisis of faith in the project. Consequently, a strong plan in place and expertise to call on have shown key in the recovery process.
However, there aren’t a lot of resources out there on recovering from hacks, and the knowledge that does exist is locked away in the minds of a few battle-tested experts.
We’re unlocking some of that knowledge here in this post-hack playbook. But first, it’s important to talk about step 0, the most effective of them all.
Pre-Hack: Preventative Measures
While it’s possible to recover from a hack, it’s better never to have been hacked in the first place. Prevention, wherever possible, is the key. And there’s lots you can do to this effect.
First, it’s essential to have automated monitoring on your contracts. Automated monitoring can help alert you to potential hacks and exploits. OpenZeppelin’s Sentinel product works well here, though there are others. Alternatively, your project can also run its own node for monitoring purposes, but that’s more time intensive.
Another basic step is to get audits for every major commit you make to your repository. Whenever you’re pushing impactful new code, you want lots of eyes to review it. In an ideal world, that code would be formally verified as well, but few teams have the skill or resources necessary. Definitely an expertise to cultivate internally if you’re able.
You also want general security procedures set in place. When things go wrong, what’s the plan? Who do you talk to? Who is in charge, what are people’s roles? How do you get in touch with everyone? What are the immediate next steps?
Finally, your project should have a bug bounty with a reward as large as possible to persuade blackhats to go white. No matter how good your auditors are, it’s possible that some vulnerability will slip through the cracks. Or, with formal verification, some rules might not have been included in the run. Despite having done your best to secure your protocol, there’s probably something you haven’t fully covered. For that, a bug bounty is your tool of choice. Ideally, you want that bug bounty on a platform like Immunefi, where you’re not only getting top quality hackers who have found criticals in other protocols, but you’re also choosing a security team you can call for help when things go wrong. Immunefi is the only place where that expertise is available.
Post-Hack: War Rooms & Crisis Management
If a hack happens anyways, ideally you’ll know about it as soon as possible so you can begin war rooming. If you’re not sure how to deal with a war room, let us know; we’re happy to help.
War rooming starts with assessing what happened; have you in fact been hacked? Once you’ve assessed that there has been a breach, you should pause that contract (if you have upgradeable contracts), since you don’t know if the hacker has finished executing the attack or how much more could be at risk. As soon as the contract has been paused, you should assess the situation and staff your war room. You’ll want at least one analyst, one person on operations, and a comms lead. You may add to that list, but you don’t want too many people in the war room for security reasons. The war room is sensitive, and too many people slows things down.
The analyst’s job is to review the code and figure out what exactly happened, conduct forensics and assess immediate damage, as well as what can be done about it. After an exploit, it’s not unusual to have a bunch of copycats preparing their own attacks. In some cases, there is no additional damage possible, but if you have any doubt, flip the circuit and pause all contracts, not just the affected one(s).
The ops lead’s job is to coordinate all the activity that’s happening in the war room. This person needs to check in every 15–30 minutes with the team to make sure things are moving along with all participants. Who should fill the ops role? You want someone versatile who has some ops and technical experience and can stay cool under pressure. Usually, this is the founder or CEO. This person has to be a rock of fortitude and keep morale high when everyone — and everything — else is uncertain. Most people will struggle under that kind of pressure, especially when they feel their net worth has been suddenly hard hit.
After the analyst staunches the bleeding, the comms lead becomes the most important person. With hacks, the damage has already been done, and these hacks provoke crises of faith. A DeFi protocol lives and dies by the faith its users have in the team and code. In the worst case scenario, project leaders fail to inspire confidence and fail to organize, resulting in large scale capital flight from the protocol and a massive token price drop.
The function of comms is to restore faith by demonstrating what the protocol is doing to mitigate the hack and resolve its consequences. This starts with a clear message that a hack has happened, that the team is working on it, and that a full postmortem is coming. Users need to know how their funds have been affected, and what’s going to be done about it.
Next, the comms lead has to answer hard questions: how did the attack happen? What’s the fix? Could the attack have been avoided? What are the next steps? Is there any restitution for users? How? But it’s not just a matter of answering questions. Comms leads need to be empathetic with users and help move them from anxiety to calm. They also need to be sure of themselves and present a strong face, since there will be a lot of FUD regardless.
All of these functions need to operate simultaneously: the ops lead organizes and directs. The analyst stops the bleeding and conducts forensics to assess the impact, discover the long-term mitigation, and figure out when it’s safe to unpause the contracts. The comms lead gives a clear message to the public about what is happening and talks with key stakeholders in the community to maintain their support though the crisis.
After all these functions are in motion and the community has been made aware of the situation, proceed with publishing a clear and transparent postmortem in the following 24–48 hours, which outlines what happened, what the impact was, the specifics of the vulnerability, if a fix is in place, and how the project will prevent future attacks. The purpose of a postmortem is to communicate that you’ve identified and resolved the issue, that you’re taking the situation seriously, and that there is a plan to prevent these events from occurring going forward.
Even with the postmortem, it’s going to take time to rebuild trust. The comms lead should be actively engaged in rebuilding faith with the community, in addition to interviewing stakeholders to discover their key concerns and how they could be alleviated. They should keep the community informed about what the protocol is doing at every step and then follow through on action plans to restore confidence in the team’s ability to weather any crisis.
This playbook will help you maximize your chance of surviving a hack. Ultimately, the best way to prepare is to prevent the hack in the first place. A big bug bounty is one of the best lines of defense, so that there is always an incentive to disclose, rather than exploit. If you’d like our help in setting up your bug bounty program, and enhancing your security processes comprehensively, then get in touch.
P.S. Hackers subscribed to our newsletter are 35.8% more likely to earn a bug bounty. Click here to sign up.
