Short downtimes, extra seconds and certificate rollover
Webserver
On Friday morning our webserver stopped responding to requests, although the database and game server was fine. Due to an internal communication issue, the webserver did not get answers from our build server, and filled up its queue waiting for responses.
Stendhal is completely open source and contributions are always welcome. In order to ensure high quality, we use Jenkins to execute automatic tests on the development version of Stendhal four times a day. Although you can look at the statistics at https://stendhalgame.org/jenkins, Jenkins runs on a home computer in order to prevent lag in game.
We fixed the webserver configuration, so that it will not wait indefinitely for responses from the build server anymore. So it will not fill up its queue and will not go offline, if this communication issue happens again in the future.
Extra Second
Tonight at exactly 2:00 o'clock server time, or 0:00 o'clock universal time, the game server started to show issues. There were no errors in the logfiles and the database seemed fine. But restarting the server process did not help. At that time – in the middle of the night –, we did not know what was wrong. We ended up restarting the complete server and that somehow fixed the issue.
Not knowing what causes issues is bad. It prevents taking counter measurements. Luckily this morning, we learned the cause: A lot of servers all over the world stumbled upon an extra second tonight, which triggered a kernel bug.
Certificate Rollover
At the moment we are busy preparing the next release of Stendhal. You may remember that once a year the updater does not work and everyone needs to do a full download because the webstart certificate is only valid for one year.
You may want to read last year's announcement for the background of the certificate issue. Last year we implemented a workaround using two distinct signatures: We only use the official signature to sign the webstart starter. This signature will normally expire once a year. The starter verifies that the rest of Stendhal is signed with our own signature.
With the upcoming release we need a new starter with a new official signature. But our own signature is still valid, so you don't have to do a manual download of Stendhal. This time, the certificate rollover will be completely automatic. Webstart users just need to accept the new certificate.
So, this year the certificate change rollover should be faster, easier, and more convenient for everyone.