Hello all, we had a system outage earlier on today at around 5:15pm EDT (U.S. Eastern), we started investigating what was going on at around 5:30pm EDT. Our host machine went down (all of our hosted services went offline as a result), we required our service provider to investigate the situation and reboot it for us. Everything has recovered now, but I just thought I'd post the updates I provided about it in Discord for those of you to look at who are not joined up.



Initial Update - 6:08 pm EDT (U.S. Eastern)

Hi all, the host server for Awakening has gone down. We have contacted our service provider to get them to investigate and reboot it for us. Once we're back in we'll bring Awakening up. At this point we are at the mercy of our service provider's support team. We'll let you all know once we've got more to share.



Second Update - 6:18pm EDT (U.S. Eastern)

Well, that was speedy, looks like we're back up already. Some of our peripheral services will still be down but the live server is operational again. The launcher patch system may be unreachable for a little while, there is a little cancel button you can use to cancel the process if you need to in order to re-enable the play button. We'll get the rest of our services back up and running as soon as possible. The launcher status will be inaccurate for a while too, that relays from a status checker that lives on our web server.



Current Update - 6:31pm EDT (U.S. Eastern)

All right, I believe I have brought everything back online that should be now. If you notice any issues please let us know via a ticket so we can investigate and resolve them as soon as possible.

After a prolonged maintenance period we believe the server is finally back in action. Thanks for bearing with us through this process. I will link the previous post I made about the server here, please give it a read: Live Server Restored



The most recent issue that we brought the server down for related to a problem that was identified with missing vendor items. We isolated the cause of this to a repeating boot loop the server got stuck in when we brought it back up the last time (this was caused by external scripts misfiring). While we assumed this would of had no affect on the server at the time, upon reviewing the logs we realised that it was improperly shutting down whilst auction updates were taking place, which lead to mass deletion of vendor items. When we brought the server back up this time, we copied across a freshly extracted version of the 8am EST database backup and left it running whilst doing some checks to ensure there was no erroneous behaviour detected. We are sufficiently satisfied that the vendor issue will not recur. That being said, if you notice anything out of the ordinary going forward please let us know via a ticket or the admin email. Server stability is our utmost priority!



As stated previously:

In the end we had to rollback to our 8am EST database backup, this means approximately 6 hours of player progress has been lost. Unfortunately there is no way to recover any player data from that 6 hour window and XP, loot and any other progress lost will not be regranted by staff. To try and make up for lost XP due to the rollback and to apologise for the server downtime we have increased the server XP multiplier to 3x the base rate, this will remain active until Monday's server restart.

This still stands true. If anyone has any question or concerns please feel free to reach out. Otherwise, we hope you enjoy the increased XP and have some fun on Awakening!

Hi all,



Unfortunately just when we thought we had her operational, a further issue was identified. I have decided to bring the server down to limit further issues and cause for disappointment in case we need to re-copy the server database again. If I left it up and we needed to re-copy the database that would result in another server rollback. I will let you know more once we have information to share, hold tight! Incognito and/or Dirge will work to get this issue resolved as soon as they possibly can!



I apologise again for the downtime, I hope you take some solace in knowing this is even less fun for me than it is for you when we need to forcibly bring the server down for extended downtimes.

Hello everyone,



We appreciate your patience while the server has been down, we have decided it is in the best interest of our players to use one of our database backups in order to bring the server back online. The corruption of the existing database was extensive and would have taken a considerable amount of time to recover, which would have meant continued server downtime.



During the process of importing our 12pm EST backup we discovered that the database corruption was already present at that stage, so it wasn't the deadlock that caused it, it just made us aware of its presence. We are still running further tests in the background to determine the cause of the corruption in order to fix that particular issue.



In the end we had to rollback to our 8am EST database backup, this means approximately 6 hours of player progress has been lost. Unfortunately there is no way to recover any player data from that 6 hour window and XP, loot and any other progress lost will not be regranted by staff. To try and make up for lost XP due to the rollback and to apologise for the server downtime we have increased the server XP multiplier to 3x the base rate, this will remain active until Monday's server restart.



We hope you enjoy the increased XP and have fun now the server is back online!

Hi all,



We are aware the live server is currently down. We are working to restore it as soon as possible.



We have identified there is a fault in the DB and we're working to see if we can understand what the fault relates to so we can try and prevent it from happening in the future. If we can't identify and fix the fault we still have a backup from two hours prior to the deadlock which we can rollback to if absolutely necessary.



This will be a potentially lengthy process so please give us some time whilst we undertake it. We will let you know once we have more to share. Thanks for your understanding and patience!



Once the server is ready to be put back online we will set the XP modifier to 3 times the base rate as a little thank you for bearing with us through Publish 10's release.