Author Topic: Servers down for unscheduled maintenance  (Read 1433 times)

Offline Crestmoor

  • Infinity Wars Staff
  • Sr. Member
  • *****
  • Posts: 353
    • View Profile
Servers down for unscheduled maintenance
« on: August 11, 2016, 03:45:36 AM »
Hey Folks,

We've brought the servers down for some unscheduled maintenance.  One of the database tables was playing up and as a result was impacting server performance and causing intermittent issues in the game.

We've identified the issue and the team is working on resolving it and bringing the servers back up.  It may take us a few hours to do a thorough test and make sure that everything is back to normal before we enable the servers again.

Apologies for the inconvenience caused.

Crestmoor
Crestmoor the Omnimind
-----------------------------
Draft deck with Omnimind, Lanstead and War Machine = A can of WhoopAss for all occasions

Offline ORISOLVE

  • Hero Member
  • *****
  • Posts: 1480
  • In the service of The Holy Roman Empire since 2016
    • View Profile
Re: Servers down for unscheduled maintenance
« Reply #1 on: August 11, 2016, 03:46:49 AM »
Good job, Crestmoor. Now why isn't Drew telling us this?  ;D

As the CEO of Yodo1, a multi-million dollar company, and Morgan Freeman voice double, I'm sure you have more important things to do than to micromanage us peasants. Appreciate the good work.
« Last Edit: August 11, 2016, 03:53:59 AM by ORISOLVE »
Where there's a will, there's a way.


Offline Crestmoor

  • Infinity Wars Staff
  • Sr. Member
  • *****
  • Posts: 353
    • View Profile
Re: Servers down for unscheduled maintenance
« Reply #2 on: August 11, 2016, 04:03:41 AM »
Good job, Crestmoor. Now why isn't Drew telling us this?  ;D

As the CEO of Yodo1, a multi-million dollar company, and Morgan Freeman voice double, I'm sure you have more important things to do than to micromanage us peasants. Appreciate the good work.

Drew's offline right now.  He's on a different timezone to the rest of the team and given that this was an unscheduled event, I wanted to get the news out to you all sooner so I posted myself.

The IW community is very close to my heart so I called Morgan to bail on him and told him to do his own acting for the day. 8)
Crestmoor the Omnimind
-----------------------------
Draft deck with Omnimind, Lanstead and War Machine = A can of WhoopAss for all occasions

Offline Candit

  • Full Member
  • ***
  • Posts: 169
  • "I make this game, James." #Poga
    • View Profile
Re: Servers down for unscheduled maintenance
« Reply #3 on: August 11, 2016, 04:47:49 AM »
Is it there yet?
"Please cease with the bumping of this particular issue." #The_Universe

Offline Sintax

  • Newbie
  • *
  • Posts: 1
    • View Profile
Re: Servers down for unscheduled maintenance
« Reply #4 on: August 11, 2016, 04:49:47 AM »
Any news on when the servers might be back up?

Offline Crestmoor

  • Infinity Wars Staff
  • Sr. Member
  • *****
  • Posts: 353
    • View Profile
Re: Servers down for unscheduled maintenance
« Reply #5 on: August 11, 2016, 05:05:26 AM »
not yet.  will let you all know as we make progress. 
Crestmoor the Omnimind
-----------------------------
Draft deck with Omnimind, Lanstead and War Machine = A can of WhoopAss for all occasions

Offline Candit

  • Full Member
  • ***
  • Posts: 169
  • "I make this game, James." #Poga
    • View Profile
Re: Servers down for unscheduled maintenance
« Reply #6 on: August 11, 2016, 06:11:52 AM »
Sneak in a full on patch in there. o.o
"Please cease with the bumping of this particular issue." #The_Universe

Offline Crestmoor

  • Infinity Wars Staff
  • Sr. Member
  • *****
  • Posts: 353
    • View Profile
Re: Servers down for unscheduled maintenance
« Reply #7 on: August 11, 2016, 06:42:34 AM »
Hey Folks,

The servers are now back up.

Crestmoor
Crestmoor the Omnimind
-----------------------------
Draft deck with Omnimind, Lanstead and War Machine = A can of WhoopAss for all occasions

Offline Candit

  • Full Member
  • ***
  • Posts: 169
  • "I make this game, James." #Poga
    • View Profile
Re: Servers down for unscheduled maintenance
« Reply #8 on: August 11, 2016, 06:49:42 AM »
About damn time. I mean, thank you. I needed my fix.
"Please cease with the bumping of this particular issue." #The_Universe

Offline Crestmoor

  • Infinity Wars Staff
  • Sr. Member
  • *****
  • Posts: 353
    • View Profile
Re: Servers down for unscheduled maintenance
« Reply #9 on: August 11, 2016, 06:51:55 AM »
Hey everyone,

For the sake of transparency, I want to update you all on what was causing the intermittent issues, why we brought the servers down and why it took a few hours to bring them back up.

This morning, we noticed that a number of players were having intermittent issues with draft as well as pack opening so we started investigating what was the cause of these issues.  we then found out that a small query that one of our engineers ran against the database got stuck and were putting locks on a number of database tables, hence causing certain functions such as drafting a new deck or opening a new pack to slow down or fail.

Anyways, we decided to bring the servers down so that we could fix the database issue but before we could do that, we wanted to put a recovery plan in place (ie. database restore) should the database fixing fail and that's what took a few hours.

Fortunately, we were able to successfully fix the database issue so we didn't need to act on the recovery plan but given that this was an unscheduled emergency update, we felt that it was good practice to have it in place just in case.

Thanks again to everyone in the community for your patience and see you back on the battlefield! :P

Crestmoor
Crestmoor the Omnimind
-----------------------------
Draft deck with Omnimind, Lanstead and War Machine = A can of WhoopAss for all occasions

Offline Candit

  • Full Member
  • ***
  • Posts: 169
  • "I make this game, James." #Poga
    • View Profile
Re: Servers down for unscheduled maintenance
« Reply #10 on: August 11, 2016, 06:55:48 AM »
Empty as hell right now. ;_;
"Please cease with the bumping of this particular issue." #The_Universe

Offline JLazeZ

  • Full Member
  • ***
  • Posts: 107
  • Let's D-D-D-D-D-D-D-DUEL
    • View Profile
Re: Servers down for unscheduled maintenance
« Reply #11 on: August 11, 2016, 07:11:46 AM »
I appreciate the transparency. Definitely would like to see more of it.
Tullio and Ju-Lin carry Infinity Wars.

Offline Rawonall

  • Sr. Member
  • ****
  • Posts: 254
    • View Profile
Re: Servers down for unscheduled maintenance
« Reply #12 on: August 11, 2016, 11:28:20 AM »
Oh, good thing it was promptly fixed.
I really needed to get my three commons as a reward for my 16 wins run.

I hope there is track of runs ended from the last patch so you will award a "rift run reward" token containing the appropriate prize(s) to all the victims of the "only commons" bug.

And btw, if there are TWO related bugs (first: you cannot get rift run rewards or open packs or do whatever else if it would add cards to your collection, and second: you don't get the cards you deserve for prizes) and you fix one of them, pheraps it would be better to specify that the other bug is still rampant if you fix just one.

Offline GeistesblitZ

  • Jr. Member
  • **
  • Posts: 54
    • View Profile
Re: Servers down for unscheduled maintenance
« Reply #13 on: August 11, 2016, 09:10:26 PM »
...

+1000 for transparency. Never seen a company actually explain what was going on in the backend, and it sure as hell is refreshing.

Offline Crestmoor

  • Infinity Wars Staff
  • Sr. Member
  • *****
  • Posts: 353
    • View Profile
Re: Servers down for unscheduled maintenance
« Reply #14 on: August 12, 2016, 04:08:18 AM »
Oh, good thing it was promptly fixed.
I really needed to get my three commons as a reward for my 16 wins run.

I hope there is track of runs ended from the last patch so you will award a "rift run reward" token containing the appropriate prize(s) to all the victims of the "only commons" bug.


we're looking at this problem right now and trying to reproduce it so that we can pin-point what's causing it.  it seems that it's not happening everytime so it's making it a bit harder for us to identify the root cause but just want to let you know we're working on this with high priority.
Crestmoor the Omnimind
-----------------------------
Draft deck with Omnimind, Lanstead and War Machine = A can of WhoopAss for all occasions