An Inside Look at a Zwift Glitch Fix

I experienced my first Zwift glitch (say that 5 times fast!) and while it was frustrating it also made me very curious.

In October I decided to give Tour of New York Stage 3 a go. Stage 3 was a lap of the Everything Bagel. It was also scheduled just after the world would change from Watopia to New York. Great! I found on the schedule the event I wanted to do. The very first opportunity to ride it. I was curious though… I live in California and my world wouldn’t technically change to New York until hours after the event happened. I figured once I joined the event it would plop me down into New York. It did not.

There I sat, in the starting pens of Watopia, with 400 of my closest friends. Much chatter was had about whether the countdown to the start would find us magically transported to the correct world. “00:03…00:02…00:01…Enjoy your ride!” In WATOPIA! What? I was annoyed, people were commenting, we all felt put out. I finished the Watopia ride, a bit flustered, and immediately jumped on Zwift Riders on Facebook to see what everyone else was saying about the mess-up.

In the comments of another’s post I found someone who said, “Jordan from Zwift sent out a message saying they apologize for the mistake and that you get the jersey anyway.” Sweet! But wait. Who is Jordan? And how did he find out about the mistake? And how did he fix it? So many questions!

So I caught up with Jordan Rapp at Zwift HQ and found out all the juicy details.

Who is Jordan?

Jordan Rapp is a Game Designer at Zwift HQ in Long Beach, California. He first heard about Zwift when he was a pro triathlete increasing his training on Zwift in the winter of 2017 as California was hit hard with rain. Jordan was also the CTO at Slowtwitch and would end up occasionally interacting with Jon Mayfield there. One thing led to another and Jordan eventually retired from triathlon to start work at Zwift.

When I asked what a “Game Designer” at Zwift does he joked that he “designs the game.” He went on to explain that his title is somewhat vague as he doesn’t see himself as a typical video game designer. Within this title he ends up working mostly on content within the game–workouts, events, training plans and routes. Here’s an example of what that looks like:

“The Everything Bagel route covers (almost) every section of road in New York. It starts out heading south and then goes up the NY KOM Reverse then traverses north along the sky roads before dropping back down, etc. I was the one who built the actual “flow” of the route – go north or south to start? Go up the KOM in reverse or forward to start? Which way to descend back into park? Which way to ascend back up to the sky roads?”

More about the Glitch

Jordan, who is responsible for in-game events, often checks in when a big event or event series is happening. Between watching the lines of communication surrounding an event and the use of Slack at HQ he often sees or hears about problems as they are happening.

In my case, he found that the event hadn’t been properly entered as a world swap. A simple human error. Once he found out about the error he fixed the code. Because it was on a late join event he then logged in as a ride leader and let people know they could end their ride and log back in as it would then take them to the New York City world. They also made right with those 400 people and gave them credit for Stage 3 even if they only rode the 6+ miles in Watopia. I appreciated that!

What about other Glitches?

Jordan made mention that ZHQ has now made changes so this kind of error won’t happen again. But I wondered about other errors. He said that in cases like mine someone at HQ would then go through to ensure like-events won’t also have a problem. It comes down to a combined effort between customer service and the communications team.

In other cases, such as big bugs that were somehow missed by QA they’ll quickly issue a patch. If it’s a smaller bug they may wait until their next planned update. Bottom line is, it’s Jon Mayfield’s call how to handle bugs in-game or on Zwift Companion.

How do they help the unhappy?

One thing Jordan mentioned more than once is the very clear love of Zwift the employees have for it. Because they love Zwift they almost all use it, which means there’s nearly always someone from HQ riding on Zwift. That means they often see the messages people are writing, and they are experiencing the same things as the rest of us: burning real calories, sweating real sweat, eating real Clif Bars, earning real XP Points, noticing real glitches. Cool!

When it comes to bigger requests or issues within Zwift Jordan said there’s an ongoing (and growing) list of features to add or refine. A current challenge for them is that they are small, so developing a feature that may seem simple can still be a challenge if they want it done well. I appreciate hearing this as it makes me realize they do care, they are working on features, and they want it to be right! Jordan also mentioned that their real goal is to fix a mistake they’ve made without even having to be asked. I like that!

If you DO ever have an issue while riding Zwift you can open a ticket at support.zwift.com where there’s 24/7 customer service. They’re also constantly watching social media (especially Twitter and Facebook) to make sure we’re happy…often working not just to fix the problem but also offering do-overs or giving credit for unlocking something you were working towards when the issue happened.

A Secret Window Into QA

Probably my favorite information from Jordan is about QA. Here’s what he said when I asked him how they make sure we don’t see most of the bugs and glitches:

“We have an awesome Quality Assurance (QA) team. Most (probably all) of them have a ton of experience in software QA and most have specific video game QA experience. They are definitely the most unique group within Zwift. Lots of crazy action figures and video game paraphernalia. I love to spend time with QA, except if it’s because I broke something… They work really long hours trying to test pretty much every imaginable scenario they can. We have good tools – and they’re always trying to build better ones – but there’s always going to be some difference between a simulated QA environment and real people logging in from all over the world. They have a very formal process for testing all kinds of aspects of gameplay. Some of the tests they run are specific tests of new features. But they also are always running “soak” tests just letting the game run, and run, and run, and run, and run to see if they can find any weirdness.

“But of course, the best part is when things just work. We’ve got a great group of experienced engineers who actually write the code – led by Jon himself – and they do a lot of peer-based code review of their work before it even goes into a build for QA to start banging on. Code review – just having someone else look over what you’ve done – is probably the single best tool for catching mistakes. It’s not fancy. But it’s amazing what a fresh–and different–set of eyes can do.”

To learn more about Zwift’s QA team, listen to this recent interview with QA Manager Mark Mermelstein >

Is Zwift Always Watching?

ZHQ has a server team with a tool to monitor the “health of the system, which includes automated alerts for big problems.” As for the human side of things, Zwift isn’t a typical 8am to 5pm, Monday through Friday company so there’s usually someone at ZHQ riding or running at any given time. So in a way, yes, Zwift is always watching.

Helpful final remarks from Jordan

“There’s no company mandate that says ‘At least one employee shall always be Zwifting at all times.’ But it sure does seem to end up being the case that someone from Zwift is on Zwift at all times.”

“We all try to have some perspective on the literal blood/sweat/tears that people shed in Zwift.”

Thanks to Jordan for being candid about what happens at ZHQ! It helps me to know they care, they play, and they too are invested in this game we call Zwift! Click here to learn more about Jordan’s latest venture in his cycling life. We wish you all the best, Jordan! Ride On!