Zwiftalizer Networking Charts Updated

Editor’s note: the following is an update from Michael Hanney, creator of Zwiftalizer, “The log file visualizer for Zwift.” Since 2016, Zwifters have been using Hanney’s creation to analyze framerate performance as well as ANT+, Bluetooth, and network signal quality.

Read below to learn about the site’s latest upgrades, along with a fascinating and technical look at how rider data flows through Zwift’s backend systems.

What’s New?

I have updated Zwiftalizer to visualize network errors in more detail. The format of the network messages in the logs changed sometime near the beginning of 2023, so I had to make an update to report TCP disconnects correctly.

I would like to thank Rich Gammon, who raised a concern about missing data on the Zwift Forums and sponsored the update to keep my coffee habit going, and Rob Pace, who gave me great test cases.

While I was fixing the issue, I also added five new charts. Here’s a summary of the new charts and some background information on how Zwift uses TCP and UDP networking.

Three UDP Charts In Order of Severity

UDP Network Timeouts

Using the UDP networking protocol, the client expects to get data from the server at a certain time interval. A UDP timeout is recorded when the client expected to get data by a certain time but didn’t.

UDP Network Errors

This chart shows failures sending and receiving client data from the server, like your current location and heading. When the game client doesn’t get the data it expects, it will try to predict where everybody is for a short time. A few errors here are not a big deal. When the list of nearby riders goes blank, or you don’t see any other riders, you’ve had too many UDP errors (or timeouts) and the game client has stopped predicting positions.

UDP Network Connection Attempts

UDP doesn’t set up a connection before sending data, so the term “connection” isn’t quite right. The client sends and receives data packets to the destination IP address and port number without waiting for an acknowledgment. The client expects to get UDP datagrams from the server at regular intervals. A connection manager task in the background keeps an eye on the communications. This connection manager attempts to re-establish the UDP communication using an alternative server IP address or port when data isn’t received as expected. This chart plots those attempts.

Three TCP Charts In Order of Severity

TCP Network Timeouts

A TCP network timeout occurs when a connection between the client and server is terminated due to a lack of acknowledgment or response within a given time period.

TCP Network Errors

This chart shows times when TCP networking errors were logged. This includes failures to download route information, upload activity (fit) files, and upload analytics data (API and Curl requests).

TCP Network Disconnects

This chart shows times when the TCP network connection closed because the game client or server didn’t acknowledge receipt of data.

To understand these charts, it’s useful to know how UDP and TCP are different.

UDP (User Datagram Protocol) and TCP (Transmission Control Protocol) are both transport layer protocols used in computer networks, but they have several key differences:

Connection-oriented vs Connectionless: TCP is a connection-oriented protocol, which means it establishes a reliable and ordered connection between the two devices communicating. UDP, on the other hand, is a connectionless protocol, which means that it does not establish any connection between the devices and simply sends the packets.

Reliability: TCP provides reliability by ensuring that all packets are received and in the correct order. In case of packet loss or errors, TCP retransmits the packets until they are received correctly. UDP, on the other hand, provides no such reliability mechanisms and does not guarantee delivery or order of packets.

Flow Control: TCP uses flow control mechanisms to ensure that the sender does not overwhelm the receiver with too many packets at once. UDP does not provide any flow control mechanisms.

Congestion Control: TCP has built-in congestion control mechanisms to ensure the network is not overwhelmed with too much traffic. UDP has no congestion control mechanisms and can flood the network with packets if not properly managed.

Speed: UDP is faster than TCP because it does not have the overhead of establishing a connection, providing reliability mechanisms, flow control, and congestion control. However, this speed advantage comes at the cost of reliability.

Why is UDP used for gameplay?

Real-time player data is sent over UDP because it’s more important to have low latency and delay than to ensure every datagram arrives intact. UDP can send data quickly because it doesn’t have to check for mistakes or send it again. In some ways it’s like a video call, where it’s okay if the picture or sound goes out for a second as long as the call doesn’t drop. On the other hand, a file download can’t lose any data, not even one bit. If it does, the file will be corrupted and unusable. TCP is used for file downloads.

The Zwift game platform makes a lot of effort to compensate for any network delays caused by players being in different parts of the physical world. This is necessary so everyone can share the same sense of time in a single virtual world.  Massively multi-player games don’t use local servers with the lowest latency because that’s not how they work. Everyone connects to the same group of servers located on the US West coast (AWS us-west-2 region in The Dalles and Prineville Oregon). To keep these delays as short as possible, the fastest network protocol is used.

What’s happening when all the riders around me disappear? 

This is a very complicated topic. I’ll try to explain what’s going on with UDP on both the client and server sides. The Internet and AWS are two of the biggest unknowns here. Zwift can’t change those things, so I won’t talk about them.

Most likely, your device’s network, wifi, or mobile data connection dropped when you suddenly found yourself riding alone. One less likely reason is that the UDP data is being blocked by a firewall. A crowded server in an AWS data center is another possible cause.

Behind the Scenes with Zwift’s Vice President of Engineering

At the AWS re:Invent conference in 2018, I met Roberto Duarte, Vice President of Engineering at Zwift. He told me in great detail how he wrote the back end of the Zwift game system. I had a non-disclosure agreement with Zwift for a few years, but that agreement is now over. For the technical readers, I’d like to tell you what I remember from our conversation. (My memory isn’t the best, and I’m sure a lot has changed in the last five years, so don’t put too much stock in what I say.)

The backend of Zwift is made up of many EC2 Linux instances, also known as nodes. I think there’s probably one node for every thousand or so players. Each player object is kept in RAM and stores information sent from the game client over UDP. The player object has its own list of the nearest 100 riders. Five times a second, the list of “riders nearby” is recalculated for each player. Each node in the cluster talks to the other nodes in the cluster from time to time. This is how the global list of everyone online is maintained. (This part uses techniques borrowed from high-frequency trading platforms in finance). If a node goes down, a client that was talking to it will start talking to a new node and the ride will continue, but normally a client would stay connected to the same node for the duration of the activity. Rarely does a node stop working. You will sometimes be moved to a different node. In this case, you might look like you’re riding alone for a moment, but then other riders will come back into view. How long you seem to be alone depends on how much data the client has stored in its buffer. Logins happen over TCP and are handled by different servers. That’s a totally different system that I’m not going to talk about here.

When there are a lot of riders at an event and they are passing each other quickly in both directions, the riders nearby list for each player changes often. This means that each server node has to work harder to move the lists around in memory, update all the other players nearby, and their players nearby, and their players nearby, and so on. This information is sent back to every client every 1/5th of a second. There is a chance that a node might not update everyone’s list of “riders nearby” quickly enough, which could cause riders to disappear and reappear on the client’s list of “riders nearby”, or be out of order occasionally.

The player data is also sent from the client to a game server node every 1/5th of a second. A clock is on both the server node and the game client. The player data object includes both times. To make up for network delays, the game server node adds anywhere from a few tenths of a second to hundreds of milliseconds to the player’s time to make it match the server’s time. In other words, a data packet from a client with a timestamp of “now” is already old by the time it gets to the server. How old it is depends on how far away the client is from the server. The point is that the in-game time is standardized on the server side. This is important for racing and helps explain why the in-game race results and 3rd-party race results don’t always match up. This is because 3rd-party race results are made by intercepting client data from observer nodes that are not the actual game server nodes.

How To See the Packet Delay for Each Rider

The g_bShowPacketDelay variable in the config file can be set to show the packet delay for each rider in the riders nearby list. This is interesting to watch because the entry in the list of riders turns red when the delay gets longer. This probably means that the delay has reached some kind of threshold for what is considered good enough for “real time.” This is just a guess. I haven’t taken any time to unpack the UDP datagrams.

set g_bShowPacketDelay=1

Why did my ride fail to upload?

When an activity doesn’t get uploaded at the end of a workout, this is a problem with the TCP network. TCP networking is used when it’s important that all of the data going to or from the server is sent in one piece. The information still gets broken into smaller chunks for transmission, but the pieces are error checked and retransmitted if any missing bits are detected. That is the biggest difference with TCP compared to UDP. Things like activity files (FIT files) and screenshots must be sent in their entirety. 

What can I do to make my network more reliable?

There are several steps you can take to improve your network connection:

First, check that your router’s firewall settings allow all incoming and outgoing TCP traffic on remote ports 443, 3023, and 3025, as well as all incoming and outgoing UDP traffic on remote ports 3022 and 3024.

Use a wired connection instead of wireless to reduce network latency and improve network stability. This could also help your ANT+ or Bluetooth signal, since the 2.4GHz wireless frequency band can cause interference for ANT+, especially channels 7 to 11. Use the 5GHz frequency bands if you have to use Wireless and ANT+.

Prioritize Zwift UDP traffic to the device you use for Zwift by using Quality of Service (QoS) settings in your router. This will ensure that your gaming traffic gets priority access to the available bandwidth. If your router has a QoS setting for streaming video, which other people in your home might be using, give it a lower priority.  

Close any unnecessary applications running in the background on your device, such as video and music streaming services, freeing up system resources for the game. These applications can consume significant bandwidth, leading to network congestion and slower performance for Zwift.

If you live-stream your rides, you might want to use a second computer to handle the video streams, and give it lower QoS priority.

You might want to upgrade the hardware on your device, like the main CPU processor. The CPU has to constantly decrypt, unpack, pack, and encrypt network data. I think this is done on the same thread as the rest of the game engine’s processing since the game engine doesn’t appear to use multiple cores.

Finally, if you really think it’s them not you, check the Zwift status page for any outages: status.zwift.com.

Questions or Comments?

Share below!

 

---

Zwift Jersey Hunting: Updates, Fixes, and Quirks

It’s coming up on two years since I wrote an article about Leaderboard Jersey hunting, and since then there have been several updates, fixes, and additional “quirks” to Zwift. So here’s an update to that original post, starting with a quick primer on Zwift Jersey Hunting while free riding.

The Basics

Zwift has four types of jerseys you can win while riding.  They are the Route (always orange, but different than the Standard Orange Zwift jersey), the Sprint (green, except for special sprint jerseys, and the Marina sprint in France which is Orange), King/Queen of the Mountain (KQOM – red polka dot, except for special KQOM jerseys), and Special (various other jerseys that are associated with Leaderboards). In some worlds, the Route, Sprint, and KQOM jerseys can combine to make more variations – this was covered in the first article. 

What’s a Leaderboard?  Those are a list of the names/times for the previous 60 minutes for specified Zwift segments that were completed by people still Zwifting. There’s one for Overall and one for Female (which you’ll only see if you’re registered as a female). The Leaderboard also cycles to your times for the past 90 days.  It’s that list on the left side of the screen – there is a setting to make it always show, which is helpful for jersey hunting.  Otherwise, it just appears when you are near the respective Leaderboard segment.

The little dots at the bottom of the Leaderboard represent the Leaderboards in the Zwift world.  Look closely at the screenshot, and you can see a little white dot in the orange dot.  That shows you’re viewing the Route Leaderboard.  The green dot represents the Sprint and the red dot, the KQOM.  If you click either of those, the respective Leaderboard will appear.

The Updates

With the various additions of Scotland, Makuri Islands, etc, there are now 95 different individual jerseys to win (this does not include combined jerseys).  That’s up by about 25 from the previous article! The new jerseys came from the various updates to Makuri Islands, the “new” Leg Snapper KQOM to Innsbruck, and the Scotland addition.

The current tally is:

• Makuri = 23 (Countryside 9, Neokyo 7, Urukazi 6. No Specials.)
• Watopia = 16 (Specials for Epic KQOM, Jungle, Alp du Zwift, Fuego Flats, Volcano, and Titans Grove)
• France = 10 (Special KQOM for Ven-Top, the same design as Watopia’s Epic KQOM)
• Scotland = 9 (see Quirks)
• London = 8 (Special KQOM for Keith and Leith Hills)
• Innsbruck = 7
• Yorkshire = 6
• Richmond = 6
• NYC = 6
• Paris = 4 (no KQOM)

(There’s also a Sprint jersey in Crit City, but we’re focused on free rides, not races.)

In the previous article, I mentioned Richmond was among the “easiest” places to snare a jersey.  But with the update that makes the reverse Worlds an official badge route, there’s more competition.  (Also of note to this reverse route, it starts you out a little AFTER the start/finish, so it makes sense to do a U-turn back to the line and then another U-turn to get credit for the loop the first time around.)

An update I find very helpful is the HoloReplay – for two reasons.  First, some start lines are not prominently shown, and as before, always start your effort at full speed.  Having your grey ghost waiting at the start line makes it much easier to see.  The second reason is to gauge your effort as you’re trying, to get another visual measure if you’re on pace to beat the best Leaderboard time.

The Fixes

With the Makuri Islands updates, various bugs have been fixed – the automatic turning for the Flatland Route (forward and reverse) works, the Orange jersey can be worn, and the jersey icons next to your name are correct.

The automatic turning for the reverse Innsbruck Route has also been corrected.  Before it would turn you left towards the Leg Snapper instead of keeping you straight to the finish line when heading back to the city.

Combined jerseys are still not possible in France or Makuri, so that must be deliberate.  Scotland will be covered under “Quirks” – a combined jersey is (kind of) possible.

Winning the top spot for the Broad Street and 23rd Street sprints in Richmond still does not earn a jersey, so that must be deliberate as well.  And as mentioned in the previous article, the Volcano lap jersey changed to the Volcano climb jersey some time ago.  There’s a Leaderboard for the lap (either direction), but no jersey to go with it.

Lastly, knocking yourself off the Leaderboard no longer happens.  (For a few months, while wearing the Leaderboard jersey, if you rode a segment that was slower than the second place rider, your first place time would be discarded and you’d lose the jersey. Yuck.)

The Quirks

There are a few jerseys that you continue to wear no matter which direction you’re going.  That’s not a quirk, since there is no corresponding Leaderboard segment that goes the other way.  What is odd, is that if you do a U-turn, Zwift will still give you the little warning at the top that “Jerseys Are Tracked Directionally.”  Examples are Maruki’s Rooftop KQOM and Railway Sprint, Watopia’s Alp du Zwift, and Innsbruck’s Leg Snapper KQOM.

Scotland has several oddities, which isn’t surprising since it’s relatively new.  As described above, the Champion’s Sprint jersey can be worn in either direction.  BUT… surprise!  The reverse Champion’s Sprint jersey can also be won and worn (in both directions), despite not having a true Leaderboard. 

Unfortunately, I didn’t get a screenshot of actually wearing it* (trust me) – but you can see I won it in the jersey icons next to my name and I got credit for it in the post-ride Activities summary.  Besides the normal Champion’s Sprint jersey, it also shows a reverse KOM and the reverse Sprint (as indicated by the small black “U-turn” symbols in the jersey icons). 

*I had already won a KQOM, so I wasn’t initially wearing the phantom reverse Champion’s Sprint jersey (see the next Scotland quirk), until I did a U-turn and it appeared (and I failed to take the screenshot).  I then won the forward Champion’s Sprint, and it became moot – I would wear the green jersey in either direction due to the forward win.

I think, but haven’t confirmed, that the phantom reverse sprint starts at the forward Champion’s Sprint start, then after the forward finish does a U-turn, goes to The Clyde Kicker reverse KQOM finish, then another U-turn to get back to the forward Sprint finish again.  Basically, an out-back segment that included both the forward Champion Sprint and the reverse Clyde Kicker.  

By the way, I won it in a blistering 10 minutes and 38 seconds as I was poking along!  Here’s the Activities summary that also confirmed the jersey, which is where I found out my time and tried to trace out the route:

Another quirk in Scotland is that a KQOM won’t combine with a Route jersey (KQOM > Route and Sprint).  From my experience above with the phantom sprint, KQOM and Sprints don’t combine either. (Although I think I saw a combined Sprint/KOM in another Zwift Insider article.)  As can be seen in the second screenshot, Sprint and Routes obviously do combine in Scotland. 

Innsbruck’s Leg Snapper KQOM also won’t combine with the Sprint (and presumably) the Route jerseys.  In fact, unlike Scotland where the KQOM seems to take precedence, the Sprint jersey will override the Leg Snapper (Sprint > KQOM). 

It’s been hard to pin down the logic:

• In Scotland, Breakaway Brae replaced Loch Loop (KQOM > Route)
• In France, Ballon replaced Aqueduc (Sprint > KQOM)
• Also in France, Marina remained over Pavé (Sprint (orange) > Sprint (green))

It’s not always the latest Jersey won that is worn, and there also doesn’t seem to be consistency among the worlds for which jersey gets priority.  More research to do.

A frustrating quirk whose cause(s) I haven’t been able to pin down is randomly not getting credit for certain sprints.  This has happened to me more than once on Watopia’s Sprint, the Fuego Flats Sprint, Makuri’s Railway and Village Sprints, and a few others.  I complete the sprint and it just doesn’t record anything.  It’s never happened on a first try, just on a subsequent attempt.  Frustrating.

Another oddity is when trying to extend the 60-minute time duration.  I haven’t fully tested this out though.  When I’m wearing a Leaderboard Jersey, I’ve tried to extend wearing it by re-riding the segment.  If I beat my previous time, all works well.  The clock gets reset.  But if I don’t beat it, but am still faster than the second place rider, the new time isn’t counted.  I time out of the jersey based on the previous time, despite having the (new) fastest time in the previous 60 minutes. 

Summary

That’s the latest status of jersey hunting in general.  I suspect there are things I’ve missed, so if you know of anything that may be of interest or benefit to others, please put them in the comments. 

Meanwhile, I’ll be keeping myself motivated in part by continuing to try to win a jersey on each free ride I do, and winning all of the available jerseys (two to go as of this writing).  Happy Zwifting!

Questions or Comments?

Post below!

---