Serverless Diary: A Frugal Agile Architect’s Approach to Logging
I. Introduction
“Logging is like a flashlight in the dark. It helps you see what’s going on in your application.”
In one of my previous blogs, I mentioned the importance of logging in distributed architecture on the public cloud. This becomes more critical as businesses wish to go faster by adopting an “as-a-service” approach while making strategic decisions around architecture choices and which platforms can help them go faster. The goal is for technology to be focused mostly on developing the differentiating factor in the market instead of investing time in building the core base, which, although valuable, doesn’t provide much direct business benefit.
The real business benefits come when actual products and features are released and consumed by end users. Companies benefit from receiving feedback from users and iterating their products quickly to provide a great customer experience.
Logging architecture and design may appear less interesting to some businesses but significantly impact the customer experience. Good logging enables reactive and proactive monitoring, ensuring higher uptime and availability for the product. It also allows businesses to learn more about where failures are happening more frequently, highlighting areas for improvement.
II. The Frugal Agile Architect Approach
Now, what would a frugal architect do if starting a greenfield project with a lot of potential for future growth? Just like any decent modern architecture, there is a desire first to learn more about the product’s impact on users. This technique is nothing new and has been used successfully in the last two decades, from big tech firms to small startups.
One architecture approach is to make big upfront decisions on platforms, like which SaaS, iPaaS, and FaaS to use. This may extend to deciding which logging and monitoring solution to implement upfront. But for many architects and projects, a large upfront investment into a product like Splunk, or Datadog may not be justified, especially in an initial slow, experimental rollout targeting a small group of pilot users.
I believe in such cases an architect has the choice to defer difficult or expensive decisions to a later point when there is more data to make an informed choice. This includes understanding metrics like the product’s growth trajectory, release frequency, number of supported users, and the importance of sustainability to the business. Picking up from my previous post discussing the benefits of cloud-native architectures leveraging SaaS and iPaaS platforms already on the public cloud, this approach provides more options to the architect.
In this example, if using Salesforce, MuleSoft, and AWS, there is no need to invest upfront into Splunk or Datadog. Instead, a more serverless, cloud-native approach with a pay-as-you-go model aligned with sustainability principles can be used. This still provides the benefits of centralized logging and observability without the large, irreversible/expensive upfront investment in another product.
Having more options as an architect is a good thing. If an option allows deferring expensive decisions to later, there is great value for the business.
III. A Cloud-Native Observability Solution
Let’s look at a specific use case. Assuming the use of Salesforce, MuleSoft CloudHub 2.0, and AWS, what could a cloud-native centralized observability solution look like?
The above design demonstrates leveraging the power of the cloud by connecting Salesforce (via CDC) and MuleSoft Cloudhub2.0 to AWS CloudWatch privately. This enables the use of CloudWatch Insights, alerting, and other features. Since security and performance are important in the cloud, a good implementation would use AWS PrivateLink via AppFlow, and EventBridge to connect Salesforce and AWS Transit Gateway to connect MuleSoft as discussed in a previous post.
“Simpler non-native connectivity using appropriate authentication mechanisms over the internet is also an option between these platforms .”
IV. Final Reflections
3 Key Takeaways:
1. Leverage more cloud-native features — they are cost-effective, sustainable, and pay-as-you-go.
2. True Agile evolutionary architectures allow expensive decisions to be deferred to later with a low cost of reversal. No architecture choice is free, but it can come at a low cost, allowing investment in more critical areas.
3. As SaaS and iPaaS platforms evolve, work with providers to learn their product strategy. For example, it would be better if Salesforce and MuleSoft supported AWS integration in a similar way instead of one using PrivateLink and the other Transit Gateway.
Gotchas to be aware of:
- At the time of writing, the MuleSoft Log4j connector does not natively support using IAM roles for connectivity, instead uses aws accessKeyId and secretKey. This implies integration traffic would not automatically route through Transit Gateway. A custom plugin/connector assuming an IAM role would need to be developed for private connectivity over the AWS transit gateway.
Public cloud providers like AWS innovate at an incredible pace. Architects should explore more native options as cloud providers enrich faster than traditional vendor products, likely due to economies of scale and rapid customer feedback. A frugal agile architect can leverage cloud-native options to implement critical capabilities like observability in a low-cost, flexible manner, deferring expensive investments until data and product traction are justified. This approach aligns well with modern product development.