As you probably noticed, for the first time in over 11 years, our site was down for 3 days. The problem looked simple at first. It seemed one of many of our data servers had a hardware issue and crashed. These things happen occasionally and we were prepared for such occasion. The estimated downtime was 1 hour. This was last Wednesday, at 4pm.
After several attempts to reboot we were faced with the unthinkable. Our site was down for 5 hours now, it was getting late, and the technical team was still in the dark. We had to think fast.
At midnight we decided to activate our disaster recovery plan. We needed to copy the latest backup of our database from our main server to a second, healthy server. While this might sound simple, when a database is as large as ours it becomes a very time-intensive operation - but it had to be done. At this point the site was down for 12 hours.
This is when we discovered that all of our backups had been destroyed. In their place, we found traces of files which had been left behind by a hacker (or group of hackers). They were in the process of deleting all of our files, and - probably by mistake - had deleted one of the system files essential for our server to run. The server crashed, but it was too late to recover the deleted data. We spent the day running recovery tools and searching for any remaining data. It was Thursday evening and the site was down for over 24 hrs.
To assure business continuity, we keep periodic off-site backups, as well as backups on the server. Our data is regularly copied from the "cloud" to an off-site storage facility, but as the copying takes very long time this is only done every four months. This would mean that we could lose at most four months worth of data. But it turns out that bad luck comes in threes: in addition to the hardware crash and the hacker attack, we were horrified to learn that the January 2013 backup did not work. Fortunately, previous backups did work.
To cut a long story short, we were able to recover our data up to September 2012. We have now resumed our standard services, and of course we will report Blackpool as usual. However, we count on your patience and understanding in the coming months as we will gradually be adding the missing competition results.
We ask you to please check your adverts and contact us if they are missing or incomplete. We will automatically extend the adverts of everyone who has been affected, and we will be providing support for updates and additions to exisiting adverts which had been lost. Logins and user names which were created in this period are gone as well. Please check if you can log in, as you may have to create your user account again if you have a problem.
This sort of attack can happen to anyone: in the past, bigger companies than us have suffered from similar attacks, as have goverments and military institutions - no one is ever safe from a sufficiently determined hacker. We do not know why we were targeted in particular, perhaps some hacker group was practicing on our server before moving to other targets. Please be assured that we have no reason to believe that any of our secure data was compromised, as this was behind another layer of security which shows no signs of being broken into. The hackers simply deleted the data.
While we know no one can ever fully protect themselves from a hacker attack, be assured that we will take this intrusion very seriously, and are taking every precaution to not allow this to ever happen again, as well as re-evaluating our backup strategy.
