• Disclosure
  • Privacy Policy
  • DMCA Policy
  • CCPA
  • Medical Disclaimer
Wednesday, June 7, 2023
SLC Metro News
  • Home
  • News
  • Business
  • Technology
    • Crytpocurrency
    • Gaming
    • Gadgets
  • Sports
  • Health
  • General
    • Business Services
  • Travel
  • Press Releases
  • Popular
No Result
View All Result
  • Home
  • News
  • Business
  • Technology
    • Crytpocurrency
    • Gaming
    • Gadgets
  • Sports
  • Health
  • General
    • Business Services
  • Travel
  • Press Releases
  • Popular
No Result
View All Result
No Result
View All Result
Home Technology Metaverse

Metaverse Platform Roblox Adds Data Center To Address 73-Hour Outage – Data Center Frontier

metaverse-platform-roblox-adds-data-center-to-address-73-hour-outage-–-data-center-frontier
Share on FacebookShare on Twitter

You are here: Home / Featured / Metaverse Platform Roblox Adds Data Center to Address 73-Hour Outage

By Rich Miller – January 24, 2022Leave a Comment

Metaverse Platform Roblox Adds Data Center to Address 73-Hour Outage

Online gaming and metaverse platform Roblox is expanding its infrastructure in the wake of a 73-hour outage in October that left its 50 million daily users offline.

The company will add a data center and expand its availability zones, hoping to address the October downtime. The company’s CEO said a key factor in the outage was “the growth in the number of servers in our data centers.” The downtime cost Roblox an estimated $25 million in lost bookings, the company said.

The outage was reviewed in an incident report released last week, which outlined how several software services contended for resources, making it harder to diagnose a bug in a database. The incident illustrates how the growing complexity of online applications can sometimes make it harder to trouble-shoot automated infrastructure, leading to lengthier outages.

The Roblox downtime was one of several extended cloud-level outages in 2021, which are driving a renewed focus on reliability engineering for complex infrastructures. DCF highlighted this issue in our 2022 Forecast, noting that “uptime is becoming more complex, requiring backup and failover strategies that span cloud, colo, on-premise facilities and edge infrastructure.”

The expansion by Roblox also underscores how metaverse-style applications will rely on significant amounts of infrastructure – a reality that could generate additional demand for digital infrastructure like data centers and network connectivity, as was noted in our recent Data Center Executive Roundtable (The Metaverse Will Need A Lot of Data Centers).

The Challenges of Growth

Roblox is an online platform for games and virtual experiences, which is available across multiple OSes and devices. Along with Minecraft and Fortnite, it has been cited as a early example of a metaverse – a collection of virtual worlds, landscapes and characters available through an immersive online environment.

Roblox is free to play and download, but operates a large in-world economy based on the Robux currency. More than 9.5 million developers have deployed games and apps, and can make money by selling items (such as clothing or avatars) in online storefronts. Roblox went public through an IPO last year, and had $509 million in revenue in the third quarter of 2021.

That’s why the lengthy outage in October became a significant business event for Roblox, which reported that the outage led to $25 million in lost revenue, and prompted the company to make $6.8 million in credits to developers as compensation for lost sales.

“This was not due to any peak in external traffic or any particular experience,” founder and CEO David Baszucki wrote on Oct 31. “Rather the failure was caused by the growth in the number of servers in our data centers. The result was that most services at Roblox were unable to effectively communicate and deploy.”

Roblox operates several data centers  with more than 18,000 servers, which support more than 170,000 software containers. The company uses a software suite from HashiCorp to manage its infrastructure. The software issues that contributed to the Roblox outage are complex, but one clear response was the need for more diverse infrastructure.

“Running all Roblox backend services on one Consul (service mesh) cluster left us exposed to an outage of this nature,” Roblox engineer Daniel Sturman in a detailed blog post. “We have already built out the servers and networking for an additional, geographically distinct data center that will host our backend services.

“We have efforts underway to move to multiple availability zones within these data centers,” he added. “We have made major modifications to our engineering roadmap and our staffing plans in order to accelerate these efforts.”

Within the last month, Roblox has advertised for a new data center manager position based in Ashburn, the leading cloud and connectivity hub in Northern Virginia.

Uncovering A ‘Pathological Performance Issue’

As is the case in many extended outages, the Roblox issues were difficult to diagnose due to confusion about the root cause of the problems. This passage in the incident report provides a high-level description.

“The root cause was due to two issues. Enabling a relatively new streaming feature on Consul under unusually high read and write load led to excessive contention and poor performance. In addition, our particular load conditions triggered a pathological performance issue in BoltDB. The open source BoltDB system is used within Consul to manage write-ahead-logs for leader election and data replication.

  • A single Consul cluster supporting multiple workloads exacerbated the impact of these issues.
  • Challenges in diagnosing these two primarily unrelated issues buried deep in the Consul implementation were largely responsible for the extended downtime.
  • Critical monitoring systems that would have provided better visibility into the cause of the outage relied on affected systems, such as Consul. This combination severely hampered the triage process.

For additional technical details, see the Roblox incident report.

As the service continues to improve and expand its infrastructure, it continues to experience periodic shorter outages, most recently on Saturday.

Rich head shot small1

About Rich Miller

I write about the places where the Internet lives, telling the story of data centers and the people who build them. I founded Data Center Knowledge, the data center industry’s leading news site. Now I’m exploring the future of cloud computing at Data Center Frontier.

SLC Metro News

© 2021 SLC Metro News

Navigate Site

  • Disclosure
  • Privacy Policy
  • DMCA Policy
  • CCPA
  • Medical Disclaimer

Follow Us

No Result
View All Result
  • Home
  • DMCA Policy
  • Medical Disclaimer
  • Privacy Policy
  • Disclosure
  • CCPA
  • Terms of Use

© 2021 SLC Metro News

We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept All”, you consent to the use of ALL the cookies. However, you may visit "Cookie Settings" to provide a controlled consent.
Cookie SettingsAccept All
Manage consent

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously.
CookieDurationDescription
cookielawinfo-checkbox-analytics11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional11 monthsThe cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
viewed_cookie_policy11 monthsThe cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.
Functional
Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features.
Performance
Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.
Analytics
Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc.
Advertisement
Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads.
Others
Other uncategorized cookies are those that are being analyzed and have not been classified into a category as yet.
SAVE & ACCEPT