Severe downtime on Thursday, June 29th

[This post has been updated on Tuesday, July 4th with some more technical details.]

Desk-Net has experienced its worst downtime in more than ten years of operations today. And we sincerely apologize for it.

Background

For the last several years Desk-Net has experienced significant growth in user numbers. Desk-Net system architecture, code basis and the servers we use have been updated many times over the last few years to cope with this growth.

We have come to a point where we needed to make more extensive changes to our system architecture, mainly by moving to a more modular one based on micro-services and a larger number of servers.

The conceptual work for this has largely been done and we are in the early phases of the step-by-step deployment of the planned changes. The first deployment including one of the intended improvements will happen in mid to late July 2017.

What happened tonight?

Due to very strong growth in June 2017 one of our core servers ran into repeated load issues. Instead of waiting for the planned major changes we decided to move quickly to another, more powerful server as a precautionary measure.

This was scheduled for tonight. And during the migration process something went wrong.

(In the end we managed to migrate to the more powerful server.)

We had two migration scenarios to choose from. We opted for a largely automated one provided by our hosting provider AWS in order to minimize the downtime. This process had been used by us before and always worked fine.

However, due to a massively increased size of our file system this process started to take a lot longer than anticipated. Efforts to cut the migration time short by cutting off the file system were not successful.

In the end we decided to take the other path which included a remapping of a new IP address which basically never is a really fast approach.

 

Once again, we apologize for this prolonged downtime and we will work hard on making sure this will not happen again (there are actually not just structural changes planned for the application, but also for how our technical teams are structured and how they operate).

 

Matthias Kretschmer

CEO & Founder of Desk-Net