SOE Up All Night! EQNL Team Stays Up As 2 Hour Maintenance Stretches to 8

Written by Feldon on . Posted in Alpha and Beta Testing, Game Updates & Maintenance


Last night, SOE took advantage of a smooth launch to their EverQuest Next Landmark Alpha and surprised everyone by dropping the NDA. At this point Murphy’s Law kicked in and the servers came down for a substantial patch to fix some of the bigger issues players had been facing. This patch, expected to take ~2 hours, turned into 4, then 6, then 8, and this morning, servers are still not up. Players may recall years ago that shortly after launch, the EverQuest 2 servers went down for 2 days without any explanation or communication. Players were kept completely in the dark. That was the old SOE.

Under the new SOE, Dave “SmokeJumper” Georgeson and others on the EQNL team (including Community Manager Dexella) did not pack it in and head home. Instead, they stayed up to make sure players are informed and fully in the loop. Dave has tweeted through the night, and posted a detailed explanation of some of the tech problems that have turned this update into a marathon undertaking.







and here’s that explanation he tweeted about…

Here’s a wrap-up. (Remember when we said we’d tell you everything? Well…here goes.)

1) When we tested Landmark internally, we only had a single world. So…we made a few mistakes, one of which was severe. Essentially, our data structure didn’t preserve the relationships between the various isles and the world that the isles were on.

2) We also had a few memory leaks. That’s pretty normal stuff that we expect to squash during testing, but in this case, it eventually caused different islands to drop and reboot. And that’s where the problems started.

3) Because the isles didn’t necessarily stay with the world they should have stayed with. Worse, that island would get orphaned out into the nothingness (not attached to a world) and when the world it was *supposed* to be attached to detected its absence, well…it made a new island from scratch instead. (And yes, that’s why peoples’ claims were disappearing.)

4) Once we realized that, we started figuring out a way to transition the current linkages to a new system that would remember the world it was on.

5) But then about half the islands came down at once (due to memory leakage). And we realized that things were going to get VERY messy if we let them come up again. So we outaged everything at once, and the game came off-line for the first time.

6) Then we started trying to splice together the data we had with where it was supposed to go. And yes, this was exactly the spaghetti mess that it sounds like. In fact, it was so bad that we’re now giving up on it (after trying to make it work for six hours, so you *know* we wanted it to work), because no matter what we do there will be some data loss, and it’s going to be a real issue to try to tell which players are bugged, which are missing data, and which are in good shape. No way for you to know from your end except that your stuff might be gone…or it might not.


Okay…so what are we doing about it? Well, we’re going to reset the voxel data so that everyone has a clean slate in that regard.

Does that mean that you’re going to lose your stuff? Are you going to lose progress? The answer is yes and no. Here’s details:

a) All voxel data will be reset. That means that the world is going to be reset back to original server conditions. Thus, if you put up a building, or dug a hole, those things are all going to be reset.

b) Things that you crafted (e.g., saw tables, stone forges, etc) WILL STILL BE WHERE YOU LEFT THEM. But that means that if you dug a hole into the side of a hill and put your forge in there, then your forge is now going to be buried inside that same hill. You’ll need to dig it out.

c) You will not lose any resources or inventory items.

d) You will not lose your claim. It will still be wherever it was previously.

So, we apologize for any building creativity that you lose from this first day’s wipe. This is not something we intended. We hope to not wipe data again anytime soon. (But you might try to save templates of the things you build in the near future, just in case. That way you can restore it if we find it necessary to wipe again.)

Make sense? If not, please ask questions!

– Dave.

Trackback from your site.

Comments (2)

  • Alawi


    I have to say in very impressed with the way SOE and the team, headed by Smedley and Georgeson dealt with this: Transparent, candid and efficient.


  • Striinger


    Some people griped about the changing targets but by the end of MY night most were like me; thankful and impressed.

    I’m not sure of the timestamps on the twitter feeds, but they were still going when it hit 11p my time on Sunday night. Correct me if I’m mistaken, but that’s around 5am in San Diego on Sunday morning. They pulled an all nighter to patch the alpha servers? Wow!


Leave a comment

- Name (required)
- Website (optional)
Please post your comments without flaming or insulting other players or personally attacking SOE employees. Comments from bogus e-mail addresses may be deleted. If you wish to have an Avatar picture, feel free to create an account on Once you sign up for a Gravatar account, then any Wordpress-powered blog you comment on will automagically show your avatar.

Powered by Warp Theme Framework