All-in on Serverless: Fixing Developer Experience at Melio

We have a saying at Melio : We are all-in on serverless.

Ever since its founding, Melio's product ran primarily on serverless. Even now, practically all of our workflows and compute is served by AWS Lambda and other serverless offerings.

Serverless has been a boon, and we've reaped the benefits, but like any technology, it has its challenges — and one of the biggest challenges we've had to tackle was developer experience.

The main challenges boiled down to:

• Production setup was often very different than the ones used locally
• You had to run a lot of local services you never change
• Infrastructure changes were common, yet managed outside the team
• Debugging a Lambda inside a complex serverless flow is non trivial

Eventually, we ended up with a process and a set of tools we are happy with:

A personal AWS account for every developer, continuously updated with the latest of all our services, each operated and owned by developer teams, that can be easily debugged locally.

Personal account for every developer

Every new developer at Melio gets their own AWS account with the main services of Melio's product already deployed in the region closest to where they work.

Challenges:

In order to do that, we had to first set up a process to manage hundreds of environments — making sure new ones have the low-level infrastructure they need (security roles, VPCs, control-tower, etc.), and updating it across all accounts when we need to make changes.

We do this using Terraform and env0, which makes it easy for our DevOps to keep track of all of the accounts.

We also had to find programmatic solutions for AWS services we use that don't scale to zero or for patterns that would be expensive to run:

• Databases that don't scale to zero (RDS, Mongo) have an instance per service in production/staging, but a single instance with different schemas per service in development.
• Scheduled processing is either disabled in personal accounts or run at much lower intervals (an every-minute cron job might change to once a day).
• Shared secrets and configuration — KMS, secrets, and configuration are saved in another shared account (which we call shared-infra) that every personal account can access.

Results:

Developers are using real AWS services in development, which made our work much more predictable — we no longer need to rely on emulating AWS features for development or testing (whether via something like LocalStack or ad-hoc implementations).

Among other things, it gives us the confidence to adopt the newest AWS products and features without worrying about how to emulate them locally or how accurate the emulation is.

This is something that is only really financially feasible because our infrastructure doesn't cost anything while idle — except for a few exceptions such as an RDS per developer (which we hope to get rid of once AWS's scale-to-zero offering becomes good enough).

The cost per engineer is mostly static and doesn't materially increase with each additional service we deploy.

Continuously updated services

When a new version of a service is merged into the main branch of our repos, it is automatically deployed to all personal accounts, keeping the developer's environment up to date at all times.

Challenges:

We had to find a way to trigger deployments every time a change is made to the main branch.

The way we did this is by having all CI pipelines upload a CloudFormation template (and Lambda code with SAM) to a centralized S3 bucket. An SNS topic has a subscription to that S3 bucket whenever something is uploaded to it.

Every personal AWS account has a CloudFormation stack that subscribes to that central SNS and deploys changes wherever needed. You can also choose to have specific services continuously deploy from a different branch.

This also meant we couldn't really use CDK's deployment pipeline as it requires downloading your code and running it locally to deploy, while SAM just creates a CloudFormation template you can deploy directly with the AWS APIs.

To get the best of both worlds, we ended up using CDK to build a template and SAM to package it.

Results:

Developers don't need to worry about keeping other teams' services up to date locally.

Cross-team development was made easier as you can set up your local environment to be updated with changes other teams make.

Infrastructure owned by developers

Teams operate, manage, and decide on their service infrastructure. They own their own CloudFormation stacks, monitoring, and resources.

Challenges:

When a downstream team chooses their own infrastructure, it puts extra effort on upstream users of the service if the infrastructure is unusual.

To prevent upstream teams from having to know how to contact each service they use, we required each service to expose an HTTP API with IAM authentication or send events to a centralized pub/sub (SNS).

Some infrastructure decisions needed to be centralized, either because of compliance, maintenance, or financial reasons — for us, this mostly meant databases and secrets must be handled by the DevOps team.

Results:

Teams have much more flexibility and knowledge about their infrastructure. It's an internal team decision to split a sequential flow to fan-out or add a queue in the middle of a flow, for example.

Teams add their own metrics, monitor their resources, and can choose the workflow that works best for them — we have teams that have a Lambda per API endpoint, others that use a single monolith service, and others using Step Functions for a managed flow.

It also means knowledge of AWS is a core competency for a team, which, for some teams, especially those that don't change their infrastructure often, has been a challenge.

Local debugging Lambdas on the cloud

Developers can intercept requests made to Lambdas in their personal account and process them locally in their own code, allowing both debugging code in the IDE and changing code on the fly no matter what complex setup your Lambda is part of.

Challenges:

We had to find a way to make debugging locally simple and straightforward. To do that, we built a tool that automatically adds an extension layer to any Lambda function you want, that upon command, tunnels all traffic through WebSockets to your computer.

To make it easy to start with, the tool knows how to read a CloudFormation template and determine Lambdas eligible to be hijacked automatically.

Results:

Debugging and making changes to code even in complex infrastructure like Step Functions or pipelines is straightforward. You just run a command in your project directory, decide which Lambda you want to intercept, perform the same operations as you would normally, and once something calls your Lambda, it'll debug in your IDE.

You can also set up a rule to intercept only specific requests so that anything but the request you want to debug can run the regular Lambda.