Mythbusting the URF Hangover
Lunar Calendars and Overwatch
URF (Ultra Rapid Fire) is an alternate game mode in League of Legends where everything is turbo speed, as the name suggests. Champions move faster, attack faster, and cast spells faster, leading to zany and chaotic gameplay. URF originated as an April Fools joke, and the players loved it so it became a staple. Accordingly, URF creates massive engagement spikes whenever it is turned on.
For a long time, URF was believed to create a “hangover effect” wherein players would binge hard for a week, quit out of exhaustion, and permanently leave the game. Looking at the engagement charts for the first few years of URF, it looked like the red line:
Internally at Riot, leadership decided that, given this permanent hangover, URF was simply too risky to run. We might be losing millions of dollars every time. URF was still overwhelmingly popular among the playerbase, so LoL tried to appease them while mitigating the hangover with limited-time runs lasting only a couple days. But the conventional wisdom was that URF was like crack-cocaine – our players loved it and wanted more, but we had to be careful not to let them consume too much. This conventional wisdom has apparently spread throughout the industry.
This permanent hangover effect was deeply counter-intuitive to me. Sure, after binging on something, you might hold back for a week or two. But usually the binge indicated that you really enjoyed the game or TV show or whatever, and predicted that you would return in the future. Moreover, plenty of other games have fast-paced alternate game modes. I had never heard other reports of a similar effect. The analyst who did the original “URF Hangover” work even commented that later URF runs seemed to not have a noticeable effect. I decided to investigate.
It turns out that in the first couple of URF runs, two major events confounded the results – Easter 2015 and Overwatch launch in 2016. After isolating them, the true URF effect is what you’d expect; an engagement spike while URF is on, a temporary trough immediately afterwards, and a gradual return to baseline within a couple weeks.
Unfortunately I don’t have any data to share that can back up my claims. Just trust me, bro.
Spring Break and the Lunar Calendar
Spring Break creates a predictable surge of engagement for League of Legends, especially back when most of the players were students in high school or college. However, the timing of spring break is not predictable. Spring break in the Americas and Europe are usually set around Easter, though the specific date range varies between countries and school districts. However, Easter itself is not on the same date every year.
Easter is celebrated on the first Sunday after the first full moon after the spring equinox, or an approximation thereof. Moreover, the approximation of the spring equinox used in the Jewish, Roman Catholic, and Eastern Orthodox traditions also varies, which is why Passover and Western/Eastern Easter are sometimes celebrated on different dates.
Easter 2015 was on April 5th, and spring breaks usually started the week prior, on March 30th. This was all very close to URF’s launch date of April 1st. This coincidental timing created the illusion that post-URF engagement settled to a lower baseline than pre-URF. When the original analyst investigated year-over-year seasonality, they did not find spring break as a culprit because the specific date of spring break varies year-to-year. In fact, Easter 2014 was on April 20th (nice.), which probably made the 2015 vs 2014 post-URF comparison look extra bad.
To isolate the effect, I simply looked at regions that do not celebrate Easter, and thus do not have spring breaks tied to the lunar calendar. Korea, China, and Taiwan are all major LoL regions that played as much URF as everyone else, but did not see a permanent hangover from 2015 URF.
Two fun asides:
Germany and France have spring breaks on a rotating schedule. The countries are divided into 3 school zones which stagger their spring breaks. This smooths out the impact on work outages and the tourism industry. Accordingly, their “post-URF hangover” effect 2015 appeared muted relative to the US.
Easter next year is going to be on April 20th again. Nice.
Overwatch
Overwatch’s playerbase has a surprisingly large overlap with LoL. At first glance, shooters and MOBAs shouldn’t intersect too much. However, both games are competitive team games with an emphasis on coordinating large chaotic teamfights around strategic objectives. Both games use a vibrant pop fantasy/sci-fi style centered around charismatic heroes. Moreover, Riot and Blizzard have large audience overlaps as socal-based PC game publishers and a shared touchpoint to Warcraft 3.
Overwatch officially released on May 24, 2016, and it also had a public beta on May 5th. This was well after the 2016 URF run had ended, but it was still during the recovery phase of the post-URF trough. Stacking these effects creates a chart that shows a lower post-URF baseline.
The sheer magnitude of the Overwatch effect caught Riot’s analysts by surprise. Plenty of major competitor game releases had come and gone, like HotS, Smite, GTA V, and several WoW expansions. At the time, it seemed reasonable to conclude that it was more likely to be something in LoL that created the effect rather than an external factor.
Once again, I compared across regions to isolate the effect. Overwatch was particularly popular in Korea and “the West”, where we saw larger engagement drops. Analysts were in the habit of only looking at North America, Europe West, and Korea as they were our “big 3 regions”. However, when looking at regions where Overwatch was less popular, such as Brazil and Turkey, the effect was far more muted. If the causal factor was URF, why would its impact vary by region?
The Vietnam Test
What would happen if URF was on for longer? After this exonerating evidence was brought to light, we ran a test in Vietnam where we left URF on for a full two months. By the end of the run, URF had declined from its highs to a far lower level than anyone expected. It seemed the novelty had worn off, and there were a few diehards left. After URF was turned off, there was basically no hangover at all, and those diehard URF players simply switched back to the other fast casual mode, ARAM.
In light of these results, LoL has gained the confidence to run URF more frequently like we see today.
Lessons Learned
While this story is specific to LoL and alternate game modes, I think there are a few general lessons to take away.
Dev Effort Bias
Developers on the game team are focused on internal factors and often ignore external factors. Everyone is crunching away at a new feature, discussing launch plans and contingencies, etc. Naturally, when analyzing the data, the hypotheses that come to mind are those about the work the team has been doing. This mindshare bias often leads game teams to attribute outcomes to their own efforts rather than uncontrollable external factors. This usually takes place as the form of positive bias, with the dev team claiming undeserved credit, but in the case of the URF it was negative bias.
Regional Cuts as Counterfactuals
Generally speaking, if a new game feature improved some outcome, we would expect to see that effect across all regions. However, external factors usually have covariates that create larger effects in some regions than others (e.g. Spring Break and Christianity, Overwatch and low-spec computers). Whenever attributing some effect to game development efforts, be sure to check across regions to validate.
Intertemporal Substitution – Short Term Hangovers
URF does truly have a hangover effect, but it is only short-term. This speaks to a broader phenomenon of intertemporal substitution. If I binge a bunch today and neglect my chores, I will play less tomorrow to catch up on my chores. The same goes for spending money, another scarce resource. When looking for impacts, it is useful to analyze metrics day-by-day to see if there are any short-term substitution effects at play that ought to be subtracted from the total impact.
Reproducibility
Plenty of pop science “fun facts” like the URF effect are catchy because they are compelling, counter-intuitive, and easy to grasp. However, they sometimes turn out to be experimental flukes and unreproducible, as with much of pop psychology. If you hear of some crazy new theory, always try to generalize and re-identify it in another context. If you can’t, there’s something fishy with the original claim.
Dota 2 has a permanent turbo mode. Why doesn't LoL make their mode permanent too? After all, even in the extended Vietnam graph, overall playtime still seems to flatten out quite a bit higher than the baseline.