The Backend Jungle: A Short Guide for Game Executives
As a non-technical person at a studio, backends never really entered my sphere of interest. It was something I comfortably left for the CTOs and Tech Leads to worry about.
Yet as my level of seniority increased, I quickly found myself in a position where I had to approve budgets for backend investments. In practice, this meant putting my name on a dotted line without understanding the details nor the different options our studio had. Not to mention that backend costs of a successful live game were very high. Right up there with personnel and marketing expenditures.
In the realm of game development, the backend is often likened to the unsung hero working tirelessly behind the scenes, powering the intricate mechanics that make our favorite games tick. Yet, despite its critical role, the backend is frequently overlooked until issues arise, leaving developers scrambling to find solutions.
After experiencing issues due to poor backend choices and/or implementation I decided it was about time for me to understand, at least on a basic level, what game backends do and how decisions around backend solutions are being made.
To level up my understanding I might have gone bit overboard:
Had calls and lunches with a handful of CTOs and technical directors
Surveyed 30 technical leaders
Interviewed founders of 7 different backend-as-a-service companies
I’ve condensed my learnings into a few page long write up and a podcast. Hope you find either or both useful.
In the podcast with Bandai Namco’s Rafael Cordoba and Heroic Labs’ Chris Molozian, we delve into the complexities of backend services in game development, exploring the advantages, challenges, and considerations developers must navigate when selecting and transitioning backend services.
Anatomy of Backend Service
Before diving into the stuff that you as a non-technical person care most about, it’s warranted to understand the basics of what backends in games do.
image courtesy of Hathora
Backend services can be divided into four components:
Account Management: Creation and synchronization of accounts across all platforms and devices.
Game Synchronization: Netcode to ensure that all players would see the same game state at the same time, which means managing differences in individual network speeds and latencies.
Matchmaking: Creating a system that matched players together based on variables such as skill level, geographical location, and latency.
Games Server Orchestration: Managing the servers that host the games themselves. For multiplayer games this includes elements such as
a. Load balancing: Distributing player traffic across multiple servers to ensure that no single server becomes overloaded, accounting for regional latency.
b. Game lifecycle management: Ensuring that games were started and stopped as needed, and that players were matched to the right game.
c. Fleet management: Ensuring that there were enough servers to handle the load of players. Scaling up and down as needed.
To Have Your Proprietary Backend Service, or Not to Have?
Whether you build your own backend solution or use one made by a third party is a business decision first and foremost.
Integrating a third party backend solution should be the first choice for any business savvy executive. It allows you to get your games faster to the market with a proven service provider whose solution is constantly improving and evolving.
Investing into a proprietary backend solution only makes sense if the backend is the core competitive advantage of the game(s) you’re building. In practice, MMOs are the most complicated games to operate, and having a proprietary backend solution can make sense for this genre. However, making a case for other genres is much more challenging. It's essential to consider the opportunity cost; building a backend from scratch could take up to 12 months, resulting in a significant delay in launching your game, compared to using an out-of-the-box solution.
Sometimes executives find themselves being sweet talked into building a proprietary backend solution with a promise of long-term cost savings. While in theory this is possible, in reality there’s not only a considerable execution risk (internal backend team fails to build a competitive solution in-time and on-budget) but also a massive maintenance cost of operating an evolving internal backend solution with a hefty engineering team.
The fact is that third party backend solutions are economies of scale. They divide the maintenance cost among their clients and charge extra for white glove customization work.
Advantages of External Backend Services
“Maintenance is the biggest cost when you have your own backend. That’s where BaaS wins. They divide the cost of maintenance across their clients and charge for customization.”
CTO of a AAA studio
External backend services, offered by dedicated providers, boast several advantages over in-house solutions. One key benefit is speed and efficiency.
External providers have dedicated teams focused solely on backend development, allowing for faster updates and improvements compared to in-house teams juggling multiple priorities. Additionally, external services often come with robust customer support, ensuring timely resolution of issues. Another key benefit is that the more mature players are battle-tested at scale, so the game team doesn't need to worry about the risk of success.
Moreover, external services can offer scalability, crucial for accommodating the fluctuating demands of live games. By leveraging the infrastructure and resources of established providers like Amazon Web Services, GCP or Microsoft, developers can scale their games seamlessly to meet growing player bases without the burden of managing infrastructure themselves.
Challenges of External Backend Services
Despite their advantages, external backend services also present challenges. One notable drawback is the risk of service disruptions. Providers may face risks like bankruptcy, which can have a very high impact on game stability, or acquisition, which has a lower impact. Developers must carefully assess the financial viability and long-term prospects of potential providers to mitigate this risk.
Another challenge is the lack of customization inherent in external services.
While providers offer pre-built solutions, they may not align perfectly with the specific needs and vision of every game. Developers must weigh the trade-offs between convenience and customization to ensure the chosen backend service adequately supports their game's requirements.
The Decision-Making Process
“When making the backend provider selection, we never have enough time to do a proper study. Of course, we’d want to, but this is just the reality of things. In practice, we talk to a few selected backend providers based on what are our needs. We get our shortlist of providers by following the industry media and by talking to peers. Studios share a lot of information!”
CTO of an award-winning AAA studio
Selecting a backend service is a complex decision influenced by various factors. This meticulous decision-making process involves extensive research, testing, and stakeholder involvement.
Typically this leads to a shortlist of potential providers based on referrals, research, and consultations with industry peers. This shortlist is then subjected to rigorous testing, with developers evaluating ease of integration, performance, and scalability.
It is important to consider scalability and future needs when selecting a backend service. In practice, this means simulations, benchmarking, and thorough testing to ensure the chosen service can accommodate the game's growth trajectory.
Transitioning Backend Services
Transitioning between backend services is a daunting endeavor, particularly for live games. It’s pretty much like changing tires on a moving car. This is why the original decision to choose a certain backend over others is so immensely important.
Should your studio find itself in a place where they have to make the switch, a phased approach to transitioning backend services is recommended. In practice this means breaking down the game into manageable components, and gradually migrating each one. This cautious approach minimizes the risk of disrupting live games while allowing for a seamless transition.
Transitioning from one backend service to another requires meticulous planning and extensive testing for success. Studios are recommended to involve external partners to ensure that they navigate complex edge cases resulting in a smooth transition.
My Key Takeaways
I learned that CTOs gladly accept backend as a service (BaaS) over building a proprietary backend. To choose the BaaS for their studio they create a shortlist based on their specific needs. To make the process easier, CTOs rely heavily on the recommendations from their peers.
The BaaS services backed by corporations such as Microsoft PlayFab are always on the shortlists as they are perceived as ‘safe options’. Yet more savvy CTOs understand that these options can be more expensive as they lack the flexibility to make them more optimized.
I’ve personally been in a position where we had to make the difficult transition from one of these ‘safe’ backend providers. I encourage every decision maker to think about the most suitable backend solution over an adverse option with a reputable name.
Overall, throughout my discussion with CTOs, it became clear that they often made BaaS decisions in an unstructured way. They were heavily influenced by their peers and even more by their backend team rather than diligently going through different options. As a result, they were not making a business decision but a safe and political choice.
This is understandable, as CTOs have a lot on their plate. Hindsight I should have been more active in the decision-making process. Reference checking, making sure the cost estimations are waterproof and that viable options were properly assessed rather than dismissed because another choice was easier to make.
Also, don't take my word for the technology choices to make for game backends and LiveOps tooling. I hope the decision-making that Rafael and his team at Bandai give the guidance studios are looking for about the right future to choose.
I want to give special thanks not only to the tens of CTOs on the Deconstructor of Fun Slack channel who answered my questions but also to the wonderful founders at Heroic Labs, Hathora, Edgegap, Gameye, Accel Byte, Metaplay, Beamable, and Pragma.
PS: Founders and executives whose teams are making a backend solution decision, reach out! I’m happy to be your sounding board.
