Tech stack at amo

Background

We strive to build AAA premium mobile products with an emphasis on design, performance and snappiness and have a history of building those in ambitious timelines while maintaining a high level of craft and low number of bugs.

This requires making subtle tradeoffs in engineering practices between short and long term decisions, deep focus on performance whether on client (startup time, animation snappiness, wow effect) or backend (low-latency) and close to zero compromise in the quality of foundational platform and framework components.

The team

The amo engineering team is currently 20 people. Mostly generalist programmers with at least 1 strong specialty over the past few years. We believe that great software need a decent mix of generalists and specialists to cover the whole spectrum of programming practices.

We all work very closely to product and design and put a lot of emphasis on having the user at the center of all our technical reflections — pushing the boundaries of what could be (more) awesome for the end user.

We make the extra effort of defining, dividing and sizing technical projects and use Linear to communicate those with the rest of the team as well as tracking progress within the team. We value rigor, pragmatism and performance in all our technical decisions.

Having a team of seasoned programmers greatly helps in that regard. Since we are working on a single monorepo, we encourage one another to cross language boundaries when it feels right. We believe that exposure to different environments makes you a better programmer.

The stack

We’ve found success going deep in code sharing across the entire stack throughout our past experiences. As code sharing is simpler in a monorepo, we moved in that direction after many years of painful segregated code sharing practices. amo is now one single monorepo containing all projects and built using the same build system: Bazel (an open-source port of Google Blaze).

iOS

We conceived our iOS app architecture to be highly modular. To this end, we use Bazel’s extreme modular approach to build a large number of Swift libraries and modularize features and components, linked statically to the final binary. We make extensive use of Dependency Injection in conjunction with api and implementation submodules to minimize module cache invalidation and ensure proper encapsulation.

For each module, a 3rd bindings submodule takes care of providing default implementations through protocols that only final targets depend upon. This way we are able to guarantee that touching implementations will not trigger recompilation of other implementation modules, only that of the final target, which in most cases can leverage incremental compilation and ensure decent build times even with 100s of modules.

Additionally, we put a big emphasis on app infrastructure components such as Navigation, Scheduling, Instrumentation, Resource attribution and make feature development to be as laser-focused as possible while ensuring strong foundations.

We use UIKit primarily given the maturity, performance, versatility and the high level of customization of our UI components. It does not, however, prevent us from using SwiftUI when appropriate. In carefully chosen components, we'll also leverage Metal and write our own shaders to unlock unprecedented performance and graphic capabilities.

We have a very careful approach of using 3rd party libraries but choose to rely heavily on RxSwift as a communication layer between the data model and the UI (see more in Mobile Infrastructure) and Swinject for Dependency Injection while we wait for a more versatile compile-safe approach to emerge from the community (or for us to build it ourselves).

Android

Similarly to iOS, our Android architecture approach has been designed to match our needs for scalability, parallelization, and reusability. The codebase is split into a large number of independent modules built with Gradle. Our modularization strategy emphasizes on low coupling and high cohesion. Thanks to Hilt, for dependency injection, and the use of api vs implementation submodules we minimize module cache invalidation and reduce build times. We keep an eye on Bazel to maintain consistency within the project and further optimize our build system.

Based on our experience building one of the most delightful Android apps in the past, we are building foundational app infrastructure components to tackle common issues like navigation, instrumentation, resource management, attribution, etc.. We make extensive use of Jetpack libraries and leverage low level APIs to make the best use of the Android platform.

We use Compose as our main UI toolkit along with the MVVM architecture. It enables us to write less code, accelerate development and simplify UI development. In some cases we do allow ourselves to switch back to the original UI toolkit. In carefully chosen components, we also leverage Vulkan and write our own shaders to unlock unprecedented performance and graphic capabilities.

Mobile Infrastructure

One interesting choice we’ve decided to stick to throughout the years and on to this new project is to have the app infrastructure (networking, authentication, data synchronization and persistence, feature data backends, etc.) done using the same technology as the backend (in Rust, see more in Production Environment) and shared across iOS and Android.

This allows for reusing a lot of code between the backend and the app (models, validators, networking components, etc.) that we think offers the advantage of letting engineers who are used to working with data management and networking do that on the client as well as the backend. Because we use a monorepo, this shared component is nothing more than another Swift library linking to many other Rust library targets, that the final app target then depends on (Bazel makes all this rather simple).

One cool thing we’ve done is that the entire RxSwift interface communicating with Rust behind the scenes is code generated, hiding all the ffi complexities. An extra perk of using Bazel in a monorepo is that any piece of code that we deem worth behaving exactly the same between iOS and Android can be done in Rust once and have its Swift and Kotlin function interface generated, reducing further the SLOC that must be written.

Production Environment

Environment

We're on Google Cloud Platform and aim to be multi-cloud soon enough. Infrastructure as code (IaC) is used to keep our production environment versioned, secure and reproducible. Plus it enables our developers to release as fast as possible, and as many times as necessary. Pulumi is our tool of choice for now, but we're open to trying new things as the environment evolves.

Our backend architecture is designed to be rapidly adaptable. Our services are designed to work alongside one another on a single binary, as a monolith. As the system grows, we can migrate it to another deployment and scale it automatically without wasting resources or creating unnecessary duplicates (unused but instantiated connection pools, etc).

We have the freedom to move between micro-services and larger services within our infrastructure as needed. Our experience has taught us that migration will consume most of our time in the future. We’ve embraced that reality and used architectural decisions and practices that make the migration process as painless as possible.

Language

We use Rust as our primary programming language. It may be surprising, but we didn't choose Rust primarily for "performance", "memory safety" or "fearless concurrency". Those are huge perks and clearly comfort our choice, but the main reason is the unique combination of being able to iterate AND grow quickly (while keeping a sane code base).

With features such as sum types, pattern matching, strong typing, traits and functional approach, a lot of problems become simpler to tackle. Landing a large scale refactoring is not as scary anymore as the compiler has our backs. ”If it compiles, it works". Scaling, both in terms of employees as well as SLOC, without compromising the quality and core team integrity is also eased by the compiler and the tooling.

Newcomers can be confident from day 1, since the compiler checks a lot of things statically. Rust-analyzer acts as guide in the codebase and provide auto tips and tricks (through clippy) to write better code, enabling reviews based on the substance of code rather than its form.

Monitoring and Alerting

For metrics, tracing, logging, and continuous-profiling, we use industry-standards such as Prometheus, Open-Telemetry, Jaeger, and Grafana. These tools enable us to guarantee fast and performant deployments, key to provide flaw-less evolving experience to our users. We undergo heavy training of these techniques to everyone doing backend development at amo.

Databases

We’ve been using ScyllaDB (monstrously fast drop-in replacement for Cassandra) for years before amo and are still convinced by it. We learned that in cases where eventual consistency is fine, a (really) fast database is an invaluable asset in your tool-set.

When you have to deal with very large number (up to millions) of requests per second, ScyllaDB is our tool of choice. When consistency is required, we’re choosing PostgreSQL (and sometimes its heavily tuned version on GCP: AlloyDB). We’re almost always plugging it via CDC to Redpanda (10x faster Kafka-compatible data streaming platform), to provide us, via a single write, a single source of truth for all our data. For example, for some use cases, we’re leveraging it to build up in-memory storage, for simplicity, speed and near real-time caching.

This stack provides us with three different options, each of which has its own advantages and drawbacks, to meet almost every requirement we have. However, we are always searching for new ways to reach our goals — LibSQL, a fork of SQLite, is a technology that we are currently exploring.

Data Platform

We consider the data we produce as one of our most precious resources. Not only for gaining greater insights about our users and how they use our products but also to be able to run jobs that then produce structured data that we can in turn use to power data-driven features.

Our internal data and analytics platform relies primary on Protocol Buffers for payloads (we make heavy use of custom field options to annotate data with various processing hints, like privacy filters for long time storage for instance). All those payloads are pushed in Redpanda, either in their own topic or in shared topics (we use a custom envelope and a custom schema registry to handle multiple schemas in a shared topic).

We then use a mix of Apache Beam and custom Rust jobs to process those messages and for some, store them in long term Google Cloud Storage in Apache Parquet file format using windowing patterns.

Those Parquet files are processed on a daily basis via a number of Apache Beam (using Google Dataflow) or Apache Spark (using Google Dataproc) jobs depending on use-cases (we favor Java, Kotlin and Scala JVM languages for such jobs). For dashboards and knowledge databases, we rely on Big Query but are also investigating Materialize.

Build, Test, Deploy Workflow

CI is an important part, if not the most important part of our daily routine. And we chose to invest heavily in it to avoid the usual pitfalls (performances, reproducibility). Using Bazel gives us another advantage here as it provides us with hermetic and reproducible builds which allow us to heavily leverage caching (even in the CI), to achieve very short feedback loops for every developer. Buildbuddy and Github Actions are used in pair to give us flexibility and profiling across our builds.

What’s next

A LOT. We are a fast evolving company, we make mistakes and work hard to correct them. It’s very probable that some of the above choices will prove wrong in the future and the sooner we find out, the better! We are at the very beginning of our journey and past experiences taught us that the most interesting challenges are ahead of us.