Alien Worlds API Journey
THE CHALLENGES OF BUILDING A FUNCTIONAL API TOOLKIT FOR THE WORLD’S MOST POPULAR CRYPTO GAME
Progress thrives on collaboration, and APIs stand among its most influential catalysts.
Short for Application Programming Interface, an API provides convenient access to live and historical data in a form that can be efficiently consumed by various applications.
This aligns with one of the basic principles of web3 – composability, which is achieved by releasing the code as open source and encouraging others to use it in their own developments. Integrating an API into a web3 interface makes it infinitely easier to use, compared to the arduous task of extracting and analyzing raw blockchain data. It encourages the community to express their creativity by reducing the number of technical issues they need to work out. It also helps build websites that require a much smaller bandwidth, creating a better user experience. In other words, a functional web3 API benefits the DApp, the community, the users, and even the blockchain it is built upon.
Of course, this only holds true if the API is technically sound and has addressed all the issues posed by the blockchain - and this is quite a challenge.
Here’s our journey in developing an API for Alien Worlds.
THE CHALLENGES OF BUILDING A WEB3 API
Alien Worlds is the most popular crypto game, with over 240’000 unique wallets connecting daily.
The biggest difficulty in building an API for such a DApp is the processing of the enormous firehose of transactions – over 13,000,000 per day. In fact, our transaction volume is so huge that it would normally take many months of computer processing time to replay the history of blocks and gather enough data to serve through the API. What’s more, such processing would never be able to catch up with the continuously added new transactions.
This means that our API must perform many multiples faster than the blockchain.
The approach we chose for tackling this ambitious task was multi-threading. Also known as parallel computing, this method involves dividing a single large datastream into multiple smaller datastreams that can run concurrently in separate computing processes.
Our custom coordinator program would break the raw blockchain datastream into smaller chunks of data and then distribute them across many processing units, which would work in parallel. The outcome is then gathered and reassembled in an easily digestible form.
However, multi-threading is not easy.
Indeed, analyzing, fixing, testing, and debugging multi-threaded history-related code has been one of the most difficult and time-consuming work streams in our experience.
Like many builders in web3, we did not have any working reference to use as a test comparison. Moreover, any issues or bugs have typically only become noticeable at the large scale, when tracking the real production data.
Any software engineer knows that debugging any multithreading computing work is notoriously intricate, and Alien Worlds API was no exception.
In the end, we built a custom queuing mechanism for parallel processing tasks, segmented the different processes into multiple sub-processes, refactored code to be type-safe, and organized the common code into reusable libraries to avoid duplicate code or bugs across multiple APIs. We also parallelized the processing of large ranges of historical blocks across many machines and built foundational tools that the business and community can easily re-use to add new API data endpoints.
UNVEILING ALIEN WORLDS API
After going through all this trouble, what did we accomplish?
The API tool kit we have now allows anyone to extract different types of information within the three main categories: DAOs, game assets, and mining stats. This constantly updated database constitutes a foundation, upon which all kinds of query end-points can be easily built.
For instance, some of the existing end-points in DAO API include DAO’s current token balance, information on custodians, elections, voting history for a given candidate... and so on. There are some 150 data points already available through the APIs that developers can mix and match to suit their needs.
We are leveraging our API to improve Alien Worlds user experience, but it also can be embedded into a third-party browser interface. For instance, if a DAO decides it needs its own website. The API can also be used by an external developer who intends to create a game within the Alien Worlds metaverse.
In fact, community was a central consideration when designing our code and module architecture. There are currently around 20 library components that make up the suite of API tools. The reason for having so many is to enable community developers to mix and match and customize the tools for their own purposes.
For example, we have chosen to use the very popular NodeJS Express web framework inside one of our modules and MongoDB as the backing database in another. If a community member prefers a different web framework or database, they could pick and choose some of our components and mix them with their own to get the best of both.
Rather than trying to create the “perfect API” to satisfy all potential use cases, we have laid the foundation with core building blocks and tackled some of the most technically challenging aspects. In this way, our community members can create their own APIs that would suit their specific needs.
What’s more, the benefits of our API tool kit extend beyond the Alien Worlds community. Some of its building blocks, such as the History Tools, can be used for any other DApp built on WAX to capture, process, and catalog any designated blockchain contract’s activity. This is our contribution to the larger WAX ecosystem, and we are happy to help move the web3 industry forward.
LOOKING AHEAD
Our API’s functionality is already quite extensive, but our ambition is to consistently elevate its capabilities.
We intend to further improve its tools with automation and code generation. This would accelerate the pace at which code changes can be tested and deployed to production environments, minimize potential human-introduced errors, and lower the barrier to entry. In this way, more developers can add value to our API through changes and enhancements. We have documented the current API tools and provided starter kits to help community teams build their own APIs and be productive as quickly as possible.
