Last Night's Game Update Deployment AKA Scope Creep, Mental Health, and You

This is not an official staff post; these are my thoughts on recent events, what went on behind the scenes, and my role here at Rebirth.

“Give this guy whatever the fuck he wants.”
-Michaeldcullen, July 24, 2019

Four days before our first Game Update went live, the Rebirth modteam held an all-hands voice chat meeting. It had been planned another four days prior to that, on the 20th, in regards to a topic many of us were hesitant about; implementing a new Archetype. Planning had been done in OuroDev’s Portal Corps channels for some time about a Synergist Archetype, as well as practical work toward another called Duo. The earlier was, and the time of this writing is, some (promising) scratch notes on a wiki page. The latter was partially implemented, to the extent that at that time it had one primary and one secondary, and its author was seeking troubleshooting regarding a crash on level up.

We were being offered an entire, working Archetype.

The majority of the team was intrigued, if not skeptical, considering the source of this work was none other than SCoRE’s private server, the same team that developed the Sentinel. Succumbing to curiosity, and our community manager’s exuberance, we all filed into voice chat to discuss business.

We talked about balance and roles and effects and the dynamic of archetypes in City of Heroes for a couple of hours. There were spreadsheets, a prototype version of Mids’ Planner, and anecdotes about Guardian’s short-lived existence on a test shard before being shelved away into digital oblivion in favor of Sentinels.

We were impressed, and we made the decision to move forward. “Give this guy whatever the fuck he wants,” Michael stated emphatically, and so we set to do precisely that. Four days later, our first Game Update (not-so-named at the time) released, to introduce series 2 ATOs, in addition to the Event enhancements, to the general playerbase. Reception was lukewarm, mostly due to the mirroring of existing ATO pricing.

Around this time things got pretty busy with the merger from Pleiades and the Super Packening. I made small preparations here and there for our second Game Update while collecting feedback and suggestions from folks on the forum and in Discord. While there seems to be some strange stigma to the contrary, the suggestions channels are our best resource for determining what we do with the server, and we have implemented several features based on player feedback. However perception outweighs evidence in that regard.

Once that dust had settled, the first Guardian AT commit was pushed. Our team set to bringing the archetype to i24 and managing any issues that arose as a result. Content workflow at that scale was relatively undocumented, and we were routinely having to weed bad beats and false leads out of our process. I took to cloning paragonwiki to our own local wiki so that we could base a Rebirth-specific fork on it, weirdly as a “fun” project to pivot to when things got frustrating.

We also started teasing hints, subtle at first, about Guardians in Discord. Mostly shoehorning in “guard,” “resolve,” “assault,” and “compose” into otherwise meaningless conversations as an inside joke. Some users caught on to the pattern, and we had a bit of an ARG with it, culminating in the “soft announcement” of Guardian on the 24th with me asking about Peacebringer builds with screenshots featuring a Luminous Assault Guardian.

This is, unfortunately, where a series of events unfolded that ensured that it would become impossible to ship Guardian for Labor Day Weekend.

Around the time that we got Guardian working on an i24 test server, BubbleWrap had announced the long awaited Asymmetrical Costumes and Freakshow Costumes were ready for testing. How could we resist? We had the workings of a sizeable update on our hands, and Bubble’s work was solid. Once we implemented asymmetry, and set back to work polishing Guardian, some unusual issues started to cascade in our environment. Text strings broke, power effects were wrong, NPCs were using the incorrect bodies. It took days to narrow down that our issues were caused by generating bins with a branch of the OuroDev build that hadn’t been tested for such outside of BubbleWrap’s costumes.

I had wanted to get patch notes up early that week. The update was massive and I didn’t want to spring it on anyone, but I also didn’t want to commit to a Friday deadline unless I was sure we could hit it. We reverted back to OuroDev’s master build, leaving BubbleWrap’s work for a future update to save time. Guardian still had some understandable issues; being originally developed in SCoRE, some powers were based on version from i25 and needed fixing. Power icons for the Guardians’ powersets were cobbled together from various sources. The Inherent still wasn’t set in stone.

Communication between the devteam and the modteam started to breakdown as the crunch set in. I was supposed to update our Community Manager with a feature list for an announcement to be made Monday or Tuesday; I got it to him Wednesday after he had already left for work. This is when I started writing up the forum threads as well, however even now the Guardian feedback thread isn’t even populated with numbers let alone the effects of the various powers listed. Retched didn’t even know where we were with features other than “Guardians and perhaps some other shit.”

At this stage the QA server had been up for a couple of days for Rebirth mods and some select folks from the community. Contrary to what I’ve been accused of, this wasn’t to keep the AT secret or give some select few a sneak peak at coming features: the QA server was hosted on my personal desktop here at home. Some of you may remember what happened to our former sysadmin, who was well protected by enterprise-class hardware, software, and experience. I used my “dickhead figures out a Rubik’s cube” skills I picked up troubleshooting i25 servers back in April and May to set up a small server for folks to help out with testing and feedback. The personal risk was extreme given the rogue server community’s history, but I needed the input from a scale test, so I opened it up to folks I figured I could trust.

But Wednesday is when the worst decisions started getting made. I decided to hold off on Update Notes for yet another day because we still hadn’t solidified some changes. At one point Radiation Melee and Armor were going to be updated to their HC counterparts because some Guardian powers relied on those versions. Originally I said “fuck it,” but after it was pointed out to be that in several places we were violating the cottage rule, I said “oh well okay unfuck it.” Not only did we lose development time to Rad Melee/Armor sitting there untouched, we lost even more when I had to go in and manually unhook Guardian versions of the powers from the existing ones so that they had no affect on one another. Guardians also feature usage of Psionic Melee powers in Psionic Assault and Psionic Mastery; I made the call to bring in the sound effects instead of disabling those sets as I didn’t want to lose an entire Ancillary Pool to something seemingly so dumb. Stone Composition’s Eruptions had massive effects issues, we elected to keep it in to “just fix the animation.” The list snowballs, but my awareness was already gone by this point and we were fully tunnelvisioned on going live for Labor Day Weekend and 2xp.

I eventually got the Update Notes posted to the forum and they immediately found their way onto the subreddit and the Facebook group. There were a lot of “I hope this comes to HC eventually” comments we had a sensible chuckle about. “They could have had it first,” we mused, ignoring the onset of panic and feeling the pressure of the eyes of the community on what we were doing with our little corner of Paragon City. Our own community was divided, and that’s my fault. I didn’t have the numbers, I didn’t allow enough lead time, and there should have been more of a conversation. We got focused to the point of mania, and I forgot the entire reason we do this god damn thing. It wasn’t originally about Guardian, and it never should have become as such. It should have been about Rebirth, and in the panic that got lost. I posted the archive to connect to the QA server in our general chat, in retrospect more as a “fuck you” than as a “here try it.” I was frustrated that I couldn’t explain what needed explaining. I doubt anything compromised my system during that period of time, but it was stupid.

The infighting in our Discord and other communities watching from afar all served to just drive us inward and aim our attention at the update. Kheldian Power Customization wormed its way in at some point, a product of my misunderstanding how the effects for Umbral and Luminous Assault worked. SKU issues arose from enabling Psionic Melee again. Pain Focusing had a power that straight up didn’t work. Bugs would crop up and with somewhat frightening efficiency we’d swat them back down. We were so absorbed in the work and so ignorant of the outside world I almost missed the server maintenance announcements. Some folks noticed that and sent me DMs to make sure I was okay. They remained unanswered, and we worked.

Windows, of all things, brought us a reprieve by taking our server down for updates for a god damned hour and a half. We used that time to get the last minute polish on Guardian. Powers had icons, Resolve was somewhat more coherent than it had been. Stone Composition’s Eruption was a simple fix, but in the way that you know the name of that one song but it’s just on the tip of your tongue. We made the decision to axe it. In a long and grueling failure cascade it was our first mulligan. There was a celebration of sorts there.

The final client and server files were packed, tested locally, and shipped to Rebirth and to our serverteam to distribute on the manifest. We pushed the final update on the server, and some odd behavior cropped up. At 2AM. To put it simply, the elements of the new update seemingly weren’t being saved to the database. You could create a Guardian, play it for a while, log off, and it’d be gone. The fix for this is usually straightforward, but our server wouldn’t recognize the changes to be made to the database. I hand edited in the missing templates from the local environment, taking great care to ensure that the existing ones were unchanged.

Almost 4AM. The last member of the team besides myself had already dropped out to head to bed. I restarted the server, logged in, created a Guardian, logged out, and it was still there, but animations were now hilariously broken. If I hadn’t been a nervous wreck I may have been able to find some enjoyment in the situation, but it just represented another failure at that point. Badges were also granting haphazardly. I replicated the issue locally, eventually pushing yet another set of files. The animations were fixed. The badges were not.

The badge issue turned out to be the universe playing a joke on me, but in hindsight I think I would have done the same thing I decided to: at 6AM I started the process of reverting the Guardian Update. The serverteam was all in bed by this point, and the files our manifest downloaded were a cruel reminder of our aspirations for the weekend, so I had to zip up our files and pin them up in the Discord announcement channel like a tramp’s bindle.

“Run a fucking batfile to connect to Rebirth, btw there’s no Guardians.” I made the announcement much more professional than that, but that’s what I really wanted to type. Issues still came in from people connecting with the launcher anyway; luckily all the weird boogeyman voodoo is clientside. That’s where we’re at right now, an hour before my job interview for some company I’ve never heard of doing support for some SaaS nobody gives a shit about.

It’s hard not to look back at the past month and think it was a monumental waste of time. Compounding this is a divide in our community that it doesn’t deserve - every server group’s decisions being overly scrutinized by the players of every other server. We finally got the community at large to stop for a moment, look around, and take notice that there are other groups out here making cool shit. Just in time for all of them to watch us fall flat on our face as hard as possible.

And that joke the universe was playing on me? That badge issue was purely visual; we could all be playing Guardians right now.

Monday was my 35th birthday, and I spent it doing this. Fuck this job, I’m going to bed.

15 Likes

I dont know if this is helpful or not but I just want to say…At the end of the day, update or no update, I’m still just grateful to have CoH back in a form that resembles what is was about for me. I’m grateful for the community that exists around the game and I am specifically grateful to you, Pluto. I won’t pretend to understand everything that goes on behind the scenes but every interaction I’ve had with you in and out of game has only enhanced my experience and made me feel connected to this community.

9 Likes

I understand you see all this as failures. But I’d venture to say the vast majority of us in the community see this as amazing success moving us closer and closer to the next chapter in playing CoH. No one here is paying, and all of us expect hardships and hiccups. Hell I still know people who tear up when they experience something in game they haven’t seen in years. Most of us have no idea what the technical stuff means and are constantly in awe that you can figure it out at all! We’re usually just happy to play, but we’re now super stoked you’re bringing us something new and have no problem waiting a bit longer. So do what you gotta do no worries that you let the community down, mate! —Totally Rad (Radiant Journey)

9 Likes

While you didn’t touch on mental health directly, understand that my wife and I, at least, very much get it and I could certainly read it between the lines there. I’ll tell you the same thing I tell my preferred streamer: take care of yourself first. Yeah, it sucks you weren’t able to near single-handedly save this for double xp weekend, but you know what? I think (hope?) a majority of us understand that you’re just one guy trying to run a ship formerly steered by paid professionals with years of experience behind them. Get some rest, rock that interview, and de-stress! Those of us that care will still be here poking fun on the Discord, we can always reschedule events if necessary, and we’re not going to hold it against you personally.

Regarding Guardian release, I for one am quite excited about it. Sure, I’m waiting to see how numbers balance out and really getting some serious testing into it, but it is in my mind undoubtedly a far better fit for the game than Sentinel ever was or will be. I’m also confident that given time we’ll be able to get it to a state where it can be not only fun but well balanced for the environment it exists in. Whatever happens there, I’m confident that most of us will be happy to have it around. Discord room temperature was cautiously excited from what I saw at the time of the initial announcement, and I think your comments that other servers are eyeing it enviously tell the tale pretty well!

Anyway, take care of numero uno and we’ll see you on the flip side when you’re ready to deal with us ruffians again!

9 Likes

Hang in there. I won’t claim to speak for anyone else, but I am so glad that we have you, and your team, working on this server and doing your best to make it the best you can be. You’re doing what you think is right, and sometimes that can blind you, or cause setbacks, or make mistakes. It’s good to re-evaluate, but in the end, isn’t that what makes a hero? Standing for what they think is right?

That rabbit hole aside, while I do appreciate all the hard work, I also hope you will take care of yourself. Life’s hard. People are hard. Sometimes harder than they should be. There’s always going to be jerks trying to hold your mistakes over your head, even if it’s just one jerk and you see them in the mirror every day. It’ll be OK. City of Heroes is fun, and having it back is amazing, but in the end, it isn’t life or death. It’s important to a lot of people, but so are you and it’s not worth your health. It’s OK to take it easy and slow. Not that I don’t understand the excitement and the push. Learn from it, better yourself, become something greater than you were.

Thanks for your take on things, and more importantly, thanks for all your hard work, effort, and consideration!

9 Likes

Having some delays is just part of life. It happenned even on live, with a lot of paid professional working on it. The fact that we are getting updates at all, that you’ve been fxing the bugs from Live and those introduced in i24 beta, is already a miracle. You and the others have been doing tons of work, have always been honest and transparent with us and give amazing “customer support”. The community of Rebirth is amazing, in big part thanks to you all.

There’s some people that will always criticize, but i think the vast majority of us aggree with your decisions and understand why they were made. I know on live, only a small % of the players actually helped with testing, a lot just wanted to experience new content quicker.

The only “error” you did in insight is wanting to please us too much. Asymetry, new AT and all in time for Labor Day. It was impossible to know in advance if there would be bug from combining them and how long it would take to fix them.

We still got double exp event for Labor Day, the server is working and we’ll eventually get the new stuff (even if it’s in a week or a month, no worries) so everything is amazing in my eyes.
When i learned about the Guardian, i can’t tell you how excited i was. To realize that we’re actually getting brand new content, not just the continuation of what the old devs did. The game is moving foward. :heart:

7 Likes

Don’t stress out too much over it! You’ve done a great job!

3 Likes

First, thanks for telling us your story. I appreciate the honesty and courage. I genuinely do.

Second, don’t let this get down too much. I’ve worked in IT for over 25 years, and this kind of stuff is par for the course. Did I ever tell you about the time I broke a Fortune 500 company for a few hours on a Friday afternoon, rendering them unable to service customers and costing them millions of dollars? (Yeah, I got fired for that.) But both the company and I survived. This is much lower stakes than that, so relax.

Kudos to you for trying to keep it together and keep it all running the best you can.

7 Likes

We stand with the Fish head!

I’d say don’t worry, but I know from personal experience that hearing than when worried just makes things worse.

The update didn’t go according to plan, and that’s a bummer. But I’m honestly impressed with how long you kept trying to make it work. You people’s dedication to this little corner of COH is down right inspirational to me!

Take care of yourselves, and don’t let any of the noise get to you. Rebirth is amazing because of the work you all put in for us!

5 Likes

You have the backing of the community! You and a small team are in part trying to replace a very large group of programmers and developers who had an entire -STUDIO- of resources at one point. Even during Live, mistakes were made and rollback happened. Keep up the good fight, but don’t forget to pay yourself first.

3 Likes

Eh my brother and I have led a WoW guild for over a decade. He started it 2 weeks after WoW came out, and even though we have passed the torch to play CoH, it is still running strong. In all those years two big issues kept popping up that amazed us repeatedly. People believed the guild to be a democracy, and people believed that the leadership owed them something as if they were paying customers. Where in all hell did they get those notions? But there it would be, over and over. People getting their panties all in a wad over things we supposedly owed them and things we should be doing better. These days, the guild is so old and so stable that our longevity speaks for itself. A bunch of volunteers sacrificing our play time over 15ish years to keep the guild going have DEFINITELY been doing something right.

No one is a special snowflake. Entitlement is for the delusional. You don’t owe us a blooming thing, Pluto. It is impossible for you fail. This is nothing but a 100% rip roaring success. Whatever happens to Rebirth and whatever bumps might come along, it’s all just a part of this grand adventure and fully to be expected. I have gotten to play City of Heroes again when I never thought I would. I cried audibly when I first opened up the character creation screen when I thought I would never play this game again. The percentage of your player base that can see that Rebirth is a huge unmitigated success are the keepers. The ones who believe you owe them anything will hopefully move on and make their irrational, unkind demands elsewhere. Viva la Rebirth. We honestly can’t thank you enough, Pluto.

P.S., to be clear, we did seek copious amts of feedback and tried to keep everyone happy best we could. But games, servers, and guilds are still not democracies. :wink:

1 Like

Pluto, thanks for puttin in the work. Greatly appreciated.