Tired of Understanding Serverless Best Practices? Get started with Best Practices and Case Study Today!

Image for post
Image for post
Original Photo by Billy from Unsplash

1. Introduction

2. Questions

  1. Legacy retirement: Datacenter managed externally with the support contract ending soon.
  2. Manual processes: Existing processes very manual and inefficient.
  3. Data quality: Poor Data quality due to manual paper recording and repetitive data entry.

3. The Test

It was an unchallenged decision that the way forward was to move to the public cloud and stop paying for expensive contracts to run one’s data centre. The intention wasn’t to move existing applications AS-IS but to transform the user experience and provide users with modern tools like a mobile phone to carry out their daily tasks, which vastly helped in addressing issues caused because of manual processes (2), leading to bad data quality (3).

So, after understanding the problem which needed to be solved, the task was to MAKE IT HAPPEN. For this to be successful, the following requirements had to be met: (among others)

1. Build a reliable and scalable infrastructure to support modern application and its growing needs

2. Build a secure application/infrastructure (security is of utmost importance on the cloud!)

3. Engineering lead time

4. Reduce operational cost and complexity

With the above requirements, now the focus shifted to “How to do Microservices” There were many options but let me focus on the two options that made to the top. Below is a simplified comparison of how Containerisation and Serverless scored against given requirements:

Image for post
Image for post

So, the agreed decision was to implement a Serverless Microservices architecture to achieve the desired business functionality.

4. The Build

Image for post
Image for post
serverless microservices architecture

The choice of the cloud provider for implementing our serverless architecture was primarily Amazon Web Services (AWS). Let me break down the above diagram and explain the design flow from left to right:

  • The Mobile app was built using the Ionic framework and used Google firebase for Analytics. The user was required to authenticate himself against an IDP (Azure Active Directory) before he could access backend microservices. On successful authentication, the user got back a JWT.
  • The user was now able to issue a secure https GET/POST/PUT operation to interact with backend microservice passing in the JWT received in the previous step. The request went via AWS DNS Route 53 configured with a Web Application Firewall (WAF) to AWS API Gateway.
  • API Gateway was an entry-point for all backend services which passed the context token to Lambda Authoriser configured with it. The Lambda authorizer performed authorization by checking for a valid JWT and role within the claim. If authorized, an ALLOW policy was returned and cached at API Gateway.
  • API Gateway proxied the authenticated/authorized request to relevant lambda functions (written in NodeJS/Typescript) deployed in a custom VPC. The lambdas then interacted with the NoSQL data store (DynamoDB) using the vpc endpoint and responded with an appropriate response. API Gateway also provided a proxy endpoint to the S3 bucket used for uploading/downloading documents/images.
  • Few of the POST/PUT operations resulted in triggering an asynchronous process in the backend. The above diagram highlights the implementation of an event-driven pattern, which provided decoupled, highly scalable, and reliable architecture by using a combination of AWS dynamo streams, SQS, and lambdas. (more on this in a future blog)

5. The Result

Following the core agile values and incremental development approach, new features and users were added every 2 weeks with no impact or change required to adjust capacity for our serverless microservices architecture and as expected, the system scaled on-demand automatically with the increase in user base throughout the year. The full rollout of mobile features was completed in a year.
So how well did the system measured against the key requirements? Let's have a look:

Image for post
Image for post

Key Learnings and Takeaways from this experience:

  • Use API Gateway caching feature where applicable. You will observe significant performance improvement.
  • There are different default timeout intervals for API gateway, Lambdas, and DynamoDB. Ensure that timeout for DynamoDB < timeout of Lambdas < timeout of API G. As a general guideline, fail fast for DynamoDB and lambdas.
  • Recursions in lambda? It is an anti-pattern.
  • DynamoDB is a NoSQL database that is extremely performant and scalable, but it requires careful design. Always define all your access patterns upfront for a microservice (the best practice of one table per microservice) and then create indexes on the table. Once you have defined Primary and composite keys on the table, you cannot modify it without dropping the table. Although, You can add more secondary indexes later.
  • Use AWS X-ray for tracing and debugging as it simplifies identifying any issues in the entire AWS landscape.
  • Make sure there is plenty of Logging, especially with serverless and event-driven style services.
  • For interaction between microservices within VPC, prefer using a private API gateway over Lambda Invoke. Functions calling other functions is an anti-pattern. There are very few edge cases where this is a valid pattern, but they are not easily broken down.
  • Use git hooks to avoid committing secrets into source control (an absolute essential if you are coding in the open)

Wish to see how to iterate on the existing design and build a serverless website as well? You won’t have to wait long!

Written by

Digital Architect, Agile practitioner, Mentor and Fitness Enthusiast.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store