Server downtime

Message boards : News : Server downtime

To post messages, you must log in.

AuthorMessage
Profile TJM
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 25 Aug 07
Posts: 843
Credit: 267,994,998
RAC: 0
Message 1532 - Posted: 9 Mar 2010, 10:25:45 UTC
Last modified: 9 Mar 2010, 10:28:48 UTC

During this weekend the server will be taken down for O/S upgrade.
The old distro (Debian 4) will be replaced by Debian 5 amd64, since the older version has a limited support and probably slows down the hardware a bit.
It's hard to estimate how long the upgrade process will take, but I'd say that 8-12 hours minimum, because I have to do a complete backup first, so in the worst case I'll be able to rollback everything to the previous state.
M4 Project homepage
M4 Project wiki
ID: 1532 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Tom_unoduetre

Send message
Joined: 18 Sep 07
Posts: 14
Credit: 2,149,222
RAC: 0
Message 1533 - Posted: 9 Mar 2010, 16:41:36 UTC

Good luck and hopefully all goes well.
ID: 1533 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
thinking_goose

Send message
Joined: 12 Nov 07
Posts: 119
Credit: 2,750,621
RAC: 0
Message 1534 - Posted: 9 Mar 2010, 18:04:50 UTC

I hope everything goes well. I've upped the work unit cache to a whole day on a couple of machines that I expect to be running through the weekend.
ID: 1534 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile TJM
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 25 Aug 07
Posts: 843
Credit: 267,994,998
RAC: 0
Message 1535 - Posted: 14 Mar 2010, 18:15:04 UTC
Last modified: 14 Mar 2010, 20:10:44 UTC

So far everything looks fine.
The upgrade process took longer than I expected, mainly due to lots of minor configuration differences between etch and lenny.
Also, I had to rebuild the server software, so I decided to upgrade all the way to the latest trunk. As a result, the server runs on revision 20902 which is the latest one and might be unstable.
So far only one bug has been found and it's just a minor glitch on the main site - missing character encoding.

EDIT: There was a bug in work generator and it crashed soon after I restarted the server. I didn't notice but someone left me a message at google chat - thanks.
M4 Project homepage
M4 Project wiki
ID: 1535 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile TJM
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 25 Aug 07
Posts: 843
Credit: 267,994,998
RAC: 0
Message 1595 - Posted: 7 May 2010, 18:16:46 UTC
Last modified: 7 May 2010, 18:18:14 UTC

07.05.2010

The server is going down for database cleanup, extended backup, software upgrade and maintenance. Later today new ciphertext will be added to the server's queue.
M4 Project homepage
M4 Project wiki
ID: 1595 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile TJM
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 25 Aug 07
Posts: 843
Credit: 267,994,998
RAC: 0
Message 1598 - Posted: 10 May 2010, 23:26:58 UTC

11.05.2010

The server is back up after a critical database failure.
I'm not sure what exactly happened, the mysql daemon just dissapeared and couldn't be restarted because one of the ib_logdata files got damaged.
I tried to avoid restoring the DB from backup, instead I used the realtime copy provided by replication slave. Unfortunately the slave is always few minutes behind the main server, so probably there was some (hopefully minor) data loss.

Right now everything seems to be running, but if you think that something is wrong with the database, please let me know immediately.
M4 Project homepage
M4 Project wiki
ID: 1598 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote

Message boards : News : Server downtime




Copyright © 2024 TJM