How technical conservatism helps us scale faster and better

At Intercom, we’re focused on the future, and we’re taking bold steps to get there. But when we make technical decisions, we like to be conservative.

In practice, being technically conservative looks like reusing existing technologies and frameworks in our stack, or promoting tried and tested patterns and solutions. We value this familiarity because we understand that the important problems to be solved are the ones that deliver customer or business value.

Instead of evaluating new technologies and spending time on already solved operational problems that ultimately provide little customer value, we can focus on improving the product by building, releasing, and iterating on solutions.

This is the sixth post in a series exploring our product principles. Here, Waheed discusses our engineering principle “Be technically conservative”.

There are many long-term benefits to technical conservatism

This principle is best illustrated by a few examples from over the years, demonstrating how “Be technically conservative” allows us to scale fast while ultimately not being a constraint. I’ve spoken previously about our experience of designing our reporting system, where we evaluated the benefits of introducing a new datastore to our stack – Redshift. It would have meant introducing a new type of database to our system that had never been tested against production. Furthermore, we would have had to spend a lot of time building up operational knowledge, maintaining clusters in production, and dealing with unforeseen issues from operating Redshift at scale.

“We leveraged our existing infrastructure to speed up the process from an estimated six months to just six weeks”

Ultimately, we decided that a familiar datastore was better suited to the job. We leveraged our existing Elasticsearch infrastructure, which powers the majority of Intercom’s search capabilities, to speed up the process from an estimated six months to just six weeks.

The initial version of the reporting system shipped a few years ago now so we’ve had an opportunity to reflect on some of the longer-term benefits of our application of technical conservatism in that case:

We solved the customer’s problem faster

Using our existing infrastructure meant we avoided spending time familiarizing ourselves with a new datastore and dealing with all the inevitable hiccups. We were able to immediately focus on the customer problem to be solved, and shipped fast, reducing delivery time by more than four months.

We made the most of the team’s time

Our Data Infrastructure team were able to continue to focus on a small, familiar set of technologies instead of being spread across multiple technologies. As a result, they had – and still have – more time to ensure the health of our existing systems and optimize our use of each technology.

“Because our set of technologies is relatively small, improvements happen regularly”

We compounded the value of ongoing improvements

Because our set of technologies is relatively small, improvements happen regularly. The product leverages those technologies, and so the impact of those improvements is compounded across everything built on top of them. A tiny improvement can have a massive, positive ripple effect across the entire product.

More teams have had more input

Using common technologies means that more engineers and teams feel confident and empowered to work with them. We’ve seen frequent improvements to the reporting product from teams across the company rather than just one team that owns a particular part of the system.

Remember that principles aren’t rules, they’re guidelines

Principles are an incredible way to align teams and have yielded great results for Intercom. But there are times when it might make more sense not to follow them. As a company scales, there is a risk that some team members will follow the principles dogmatically or interpret them incorrectly. Defaulting to technical conservatism should not mean that we never introduce something new.

“Technical conservatism means favouring a technology that’s already in your stack – but only if it’s the best option”

Technical conservatism means favouring a technology that’s already in your stack – but only if it’s the best option. In some situations an existing technology might not be suitable. If it can’t answer the following questions, we might look further and evaluate alternatives:

• Does the new tool allow your business to scale more effectively?
• Does it allow your team or org to move faster and deliver value more quickly?
• Does it solve a customer problem that couldn’t be solved with your existing tools?

If you answer “yes” to any one of these, it may be worth considering introducing that new tool. At Intercom, there was a recent example that answered “yes” to all three questions.

Users, or our customers’ customers, are core to the Intercom platform. As we have grown, so have our customers and their needs in terms of how much user data they store within Intercom. The vast quantity of user data was leading to scaling issues with our current user datastore at the time, and to ensure we could continue to support existing and new customers we needed to rethink our current solution. That ultimately led us to introduce a new technology to our stack – here’s how we arrived at this decision.

We were scaling faster than our datastore would allow

We had been using MongoDB for roughly five years and had the operational scars to prove it. We had optimized every aspect of owning and running it – from the type of hardware it ran on, to the queries we ran on it. We were at the point of diminishing returns and were seeing strong signals that it would stop being fit for purpose within a couple of years – and might even become a bottleneck in the company’s growth.

“Regularly think about the trajectory of your business and ask ‘Will what got us here, get us there?’. This will allow you to be proactive about your choices rather than reactive”

This is where having a strong, forward-thinking technical strategy is key. At this point we had enough data to suggest that we might need to evaluate another approach and had the runway to do it. Regularly think about the trajectory of your business and ask “Will what got us here, get us there?”. This will allow you to be proactive about your choices rather than reactive, and mitigates the risks around the unknown unknowns of introducing a new technology.

Our technology was slowing down our team

At Intercom we strive to run less software. In this case, even though we were adopting a new technology with unknown unknowns, adopting DynamoDB allowed us to do just that.

Previously, we were self-managing the scaling of MongoDB along with the code for balancing load – a significant overhead for the team. Part of the draw of DynamoDB was that it was managed by the vendor, AWS. This meant that, although there was an initial cost to adopting it, ultimately it would be cheaper and save the team a huge amount of time and effort.

“Not being dogmatic about technical conservatism enabled us to replace a technology with large overheads and limited capabilities”

It may seem counterintuitive, but introducing a new technology ultimately resulted in us running less software. Not being dogmatic about technical conservatism enabled us to replace a technology with large overheads and limited capabilities with a new technology that was less burdensome operationally and more scalable.

We were ruthless about requirements

We would sometimes require MongoDB to perform complex and expensive queries that risked availability and the performance of more common but less complex queries. When evaluating DynamoDB we realised that it wouldn’t support those complex and expensive queries but it would be much better at the simpler and more common ones.

We already mostly used Elasticsearch to perform complex queries, and the need to migrate forced us to review and more deliberately define the capabilities we required from our user store and ultimately allowed us to improve performance for its primary use case: retrieving single user records.

“When thinking about replacing a technology, don’t take it for granted that you will use the new technology in the same way”

When thinking about replacing a technology, don’t take it for granted that you will use the new technology in the same way. Your requirements will likely have changed considerably over time, and the rest of the stack will have evolved or matured so that use cases become more narrow. This will open up opportunities to adopt more focused, performant technologies, or offload some technologies to be managed by a vendor.

Make your principles work for your business, not the other way around

Technical conservatism is a great tool for allowing your teams to stay focussed on what’s important – solving customer problems and delivering value without spending precious resources on answering questions that have already been answered.

However, becoming too rigid in the belief that introducing new technologies is bad could prevent you from taking advantage of technologies that will help you run less software and scale more easily and quickly. It’s important to apply this principle in a way that works best for your team, and your business, in the long term.

Are you interested in joining Intercom’s engineering team? Find out more and see our open roles here.