During the first week of January, flying back to Mauritius, I get an intriguing message from my colleague saying one of our client websites was down. We have been freelancing together for some time at Dessart Studio, building and hosting websites. It was the first time we had such an issue after over a year since deployment. My first guess was, the client might have done something on the admin dashboard, which in this case was quite unlikely. But not as unlikely as what I came to discover. I was still hopeful that was a predictable issue we can easily fix, explain to the client, a make sure it doesn’t happen again.

A few hours later, close to midnight, I get to my college campus, the student in me all excited for the new semester — at least, I hope he was, greet my roommate, put down my luggage, and, well, switch on my PC. I SSH on the server and … things had changed. Weird files appeared in that website directory, and suspicious codes were injected here and there … What happened?

Did someone SSH into the server, and do all this?

This one is quite unlikely, but still a valid potential reason for such an event. Someone, a hacker, could have got their hands on my SSH keys and party in the server. But why that particular website? Why now? How did they get access to my keys? Anyway, I checked the logs from the terminal command history and last SSH access, and nothing suspicious there. So what happened?

Let’s revert back to an older backup, rest and come back to it tomorrow!

The oldest backup kept was 7 days old. But even there, same issue. What? We had been hacked for over 7 days, and I am learning about this just now? Well, I never expected that server to be hacked in the first place. I had no kind of alert or active monitoring setup. If a website goes down, high CPU or memory usage, suspicious login attempts, or anything, I wouldn’t tell unless it’s me doing it, or I am standing next to whoever is doing that. This issue could have easily been solved if I learned about it within the 7 days cycle. The website would have been back online in no time, and I’d have all my time and peace of mind analyzing the hacked server and upgrading the security measures.

So no sleep for us. Let’s get to work, and remove everything that looks suspicious

There was a lot of it.

Random suspicious plugins

Figure 1

Base64 suspicious code

Figure 2

The index.php wasn’t spared!

Figure 3

And many other files …

I removed everything that looked suspicious and got the website back running. Changed all passwords and reported back to the client we were all good, got hacked but everything is under control — ohh, I can imagine how big tech companies feel making such big announcements. Tired, I slept, knowing everything wasn’t really okay yet. We still don’t know how that happened, and how not to let it happen again.

Checking apache access logs

This issue was hunting me, I was already back on it in the morning, exploring the logs to see what really happened. The most recent logs didn’t seem to have any suspicious activities, nothing that could have caused that security breach. I could see suspicious activities from December 26th as from the screenshot below.

Figure 4

It shows traces of the “Scanning Phase” in ethical hacking — or just hacking, this was definitely not ethical, trying multiple paths that can hold sensitive information or just be vulnerabilities.

The attacker tried paths like .env which usually holds environment variables that in some cases have database passwords and API private keys. It also checked for some other paths like /credentials in search for possible leaks of passwords or any sensitive information. There are automated tools for such scanning work, like wp-scan for WordPress websites in particular. And yes, they did get something interesting, the /rmlrpc.php returning a 200 response code, which they used a brute force attack with and likely got an admin password a few days later. From there, installed plugins that could modify files on the server and inject all the malicious code we saw earlier.

Now you’d ask, how do you get brute-forced on a website, you really couldn’t just block access after multiple attempts? That’s basic defense a 3-year-old kid must be familiar with, after blocking their mum’s phone trying to play games!

Yes, I was just as confused.

I did have a maximum attempt set for login, but it was for the /wp-login path, that form you put your username and password on the browser. I came to learn, WordPress has an API route, this /xmlrpc.php that’s not covered by the brute force protection I had. But why does it exist in the first place? You can read more about the xmlrpc.php exploit from this well-written article by Rizwan: “Wordpress xmlrpc.php -common vulnerabilites & how to exploit them”. At a high level, it allows interacting with the website backend programmatically and doing things like creating, or editing posts. I didn’t need that, and typically, I’d guess most WordPress websites don’t need that, but it’s there, active by default.

Let’s fix this brute-force issue

To fix the xmlrpc.php issue, I simply disabled access to that route as I wasn’t using it. I added the following lines to the .htaccess file:

# Stop access to xmlrpc.php
<Files xmlrpc.php>
order deny,allow
deny from all
</Files>

All good Now?

I wish! I really do, but everything becomes more interesting, and maybe a little scary. I realized, every time I load a page on the website, this file is overwritten, with the content below: making sure this path stays open. What’s happening?

I did a key search for files with “.htaccess” on it, looking for a file that might responsible for this.

grep -lir ".htaccess" ./

Guess what I found, a PHP code is well hidden in an image.

Figure 5

The code is run every time the home page loads, overriding the .htaccess file. Yes, this could have been easily avoided with good permissions set on the files, but for some reason, a few years back, I left the folder with 777 (read, write, execute) permissions for any user. A number of “small” issues that, put together, create a door for a hacker to penetrate the server and pretty much do whatever they want.

Now, the .htaccess issue is solved, we set the least required permissions for all files and directories — you can refer to this article for WordPress permissions. This applies to any system, you should always give the least required access/permissions to users. There is no “giving more permissions in case they need it in the future”. Just wait for that future to come, unless don’t mind attracting alternative futures where you have to deal with hackers 🙂

Other security considerations

After reviewing and fixing all the issues, I finally set up better monitoring and alert systems to keep an eye on what’s happening on the server. If you’re using WordPress, they are a good number of security plugins you can use, you just need to search for what’s best for your use case. Same for any other system that’s discussed in this article.

We discussed in this article the steps I went through to recover a hacked website and review the security measures to prevent a similar attack in the future. I doubt the hacker had any particular motive to target this particular website, I suspect it’s just one of those bots lurking around the internet, looking for vulnerable websites, and reporting back to a hacker if they find something interesting. Statistics from statistica on the internet traffic from 2022 suggest over 40% were bots, and a study from Imperva suggests that most of those are malicious bots. So, you need to be gatekeeping some nuclear bomb secret codes on your website to be a target, you just need to be out there on the internet. The internet is a wild world, be safe!

Nice to see you reached here! I use this space in medium to share my experience in tech. You can follow me to be updated on my latest articles. Also, feel free to use the comment section to share your thoughts! Have you been hacked before?

server articles