An Unusual Request
Some stories just can't be made up. They are so unexpected that only the real-life could write them. This story is one of them.
It started when I got back to the office after the New Year celebrations. Checking the inbox I've found one particular e-mail that caught my attention.
At first, it looked like another scam inquiry, but my gut feeling made me respond. To my surprise, I got an answer pretty quickly, but what shocked me the most was this:
The client was from Ukraine (not the typical destination for our customers) and had a unique problem.
They were running a Cryptocurrency exchange that was being continuously hacked. They didn't know what's wrong and needed to solve this problem discreetly and ASAP.
If that alone doesn't give you goosebumps, consider this. The exchange was a fun garage project built in 2015 by two students. Back then, the crypto was traded only by the diehard fans. There was nothing like documentation, tests, or oversight of who from the past developers had access to the server.
Today, 650.000+ users were trading 400+ pairs of cryptocurrencies. The volume of transactions reached $20M per day. All that was running on 5 years old code without any updates and the garage-like set server, where even basic server logs were not available.
In this mess, there was some loophole that was exploited by the hacker, whom we shall find.
Challenge accepted!
Setting up the Trap
After a couple of months of analyzing more than 140 GB of logs using various tools, we found the hacker's IP address.
Not much, we still didn't know how he exploited the system, but it allowed us to set up the trap. So we set it up and waited for his next attempt…
After 2 weeks, he struck again, and the trap triggered. Bingo, we got him!
It showed the hacker used a pretty sophisticated method that completely shocked us. He had access to the secret key that was not publicly accessible.
- Where did he get the key?
- Was it some past pissed off programmer who was not paid and got mad?
- Did they hack the testing server?
We don't know the answer to that question, and I think we never will. After we regenerated the key, the next attempts to breach the system failed. Heh, checkmate, mate! :)
Lesson Learned
We got asked If we can fix and maintain the code after that. Unfortunately, we found out that the only safe solution is to rewrite the whole system from scratch. We are still talking about that, so who knows, maybe one day we are going to develop a serious Binance competitor :-)
Our lesson from this story was that not every time common sense transforms into common practice.
Only an insane person would think it is a good idea to run transactions with volume over $600.000.000 per month on the server without basic security systems and procedures. But today everything moves so fast; you could be easily caught with your pants down :-)
