The finance industry has always been one of the most impactful sectors in our lives. In the era of the pandemic, it’s gaining even greater momentum – everyone goes digital, big finance institutions introduce aggressive digital transformation strategies, new innovative cutting-edge players appear on the market, and the list goes on. This brings a new challenge to the surface, a challenge that has always been relevant but rarely given the attention it deserves – software scalability.
Get to the bottom of the software scalability requirements
Every financial company is built upon the expectation that it will grow tremendously, reaching thousands, even millions of users. However, not every system needs to be scalable. Some systems need to support a small group of users and can still be extremely successful. At Accedia, we’ve come across multiple similar applications. Starting from digital wallets, developed for the top 1% richest clients in a bank, to B2B solutions with less than 1000 users that generate over 100M dollars daily. So, before starting a project, make sure you have the right understanding of the problem you are trying to solve. Make sure you know how the system will be used, how many customers will use it, and what are the scalability requirements. Once you know that your application will be used by millions of users, you need to build a proper strategy for addressing scalability. To do so, you first need to find the answers to some essential questions:
- What is the initial number of users that will use the application?
- What is the expected number of users?
- How the userbase will grow over time?
Having clear expectations and understanding the requirements, you will be able to design and build a solution that evolves smoothly over time and meet the projected system load.
Plan for software resiliency
Another question, related to software scalability, is also tightly connected to the resiliency of the system. Can your system handle sudden load bursts? Or, in other words, can your infrastructure automatically handle situations in which the load increases from 1M to 10M and then degrade back to 1M users? Something we’ve discovered in our practice is that some of these load bursts can be partially predicted – black Fridays, Christmas shopping, etc. Don’t forget that some of the triggers may be specific to your organization. For one of our clients, it turned out that the most active months for car insurance renewals are late December and early January, followed by the quietest times in February. So, if you know that the load of the system will change, act proactively. Don’t wait until it’s too late.
Conduct trade-off analysis
While scalability has a strong positive impact on your business performance, don’t forget that it doesn’t come for free. It may compromise other quality requirements like performance and data consistency. One of the key aspects of designing a software system is finding the proper balance between the different functional and quality requirements. The finance industry will always require a strong presence of security, history and audit trailing, performance, backup, etc. Therefore, it is important to make the necessary trade-off analysis before starting a project. It will present you with alternative scenarios and objective criteria to base your decisions on, which will later guarantee that you’ve found the sweet spot for software scalability.
Choose a hosting infrastructure that meets your software scalability needs
There is one last thing to consider before kicking off the development of your market killer software – Where is it going to be hosted? For example, running it on the cloud provides a lot of flexibility and shortens the time-to-market. Modern cloud providers give you all the tools that make your life easier – automatic deployments and scalability, tight integration among the hosting services, existing predefined services, etc. However, some financial services companies, most often big banks or insurance institutions prefer using their own infrastructure on-premises. While this increases some of the security aspects like data access, this may bring some limitations in terms of software scalability. We’ve worked together with clients using both approaches and considering the pros and cons in early project phases and planning accordingly.
Organize your teams for software scalability
Now, the development has started. You realize that some of the priorities have changed due to shifts in business requirements. What a surprise?! Such changes are part of the business, and we should incorporate them in our process to keep up with the market demand. While priorities change, it is vital to stay focused on the quality requirements, including software scalability.
A key aspect to responding quickly to changes and outperforming your competitors is organizing your teams properly. As teams increase in size, the coordination overhead increases with it. A general solution that works well is organizing the teams around business capabilities or domains. This not only boosts the development speed and teams’ autonomy but also enables you to scale much efficiently. At such a point you may start considering microservice architecture. This is the point at which the benefits of microservices start outperforming their drawbacks. Of course, microservices would require a high level of automation and monitoring. This is a key requirement for dealing with software scalability as the codebase increases and the infrastructure cannot be handled manually anymore.
Take the finance industry specifics into account
Like every other sector, the financial services industry has its own specifics. Many financial institutions prefer to keep their data on-premises spread across different geographical locations. This could introduce some problems with accessing those systems due to network connectivity or non-replicated data. To deal with it, you may need to build separate tools that ensure connectivity or aggregate data for reporting purposes. Reporting has its own role, and it is often associated with slow and heavyweight calculations. Some of the practices that have helped our clients are using extensive caching for rarely changing data. In many cases, you need to process millions of records for some operations. In this case, you could prepare some pre-calculated results. For example, mutual funds are updated once a day. There is no need to calculate the performance of the fund for every user request, while you can pre-calculate it once and use it for the rest of the day.
While more and more financial services companies introduce cutting-edge technologies and innovative software solutions, there are still many which use legacy systems at the heart of their business. Legacy systems may be a hard nut to crack. In many cases, they are the main bottleneck for software scalability and performance. And what’s more, it’s extremely hard (read impossible) to get them out of the equation. However, some best practices can come in handy. For example, instead of directly working with the legacy system, introduce a middleware. This way, you would have greater control over the middleware layer, and you could easily introduce some optimizations related to performance, scalability and many more attributes.
Implement automated monitoring and recovery procedures
Last but not least, modern financial services solutions are living systems. They are launched as small single-purpose applications and evolve constantly to become complex multi-purpose systems. This means, that you need to spend sufficient time and effort on ensuring the smooth operations of your existing systems. Some of the best practices that we’ve used for our clients are implementing strong automated monitoring and recovery procedures. These can include anything in between bringing on board SRE (site reliability engineering) teams and introducing autonomous self-healing capabilities of the systems.
Wrapping it all up
To summarize, software scalability in the financial services industry come in a great variety of flavors. All systems bring their unique challenges to address and unfortunately, there is no silver bullet. To solve them, you need to apply a system thinking approach and consider multiple factors before coming up with the most optimal solution.
Don’t know where to start or how to mix the described pillars into a single coherent solution? Learn more about our technology expertise in finance and refer to our experts for a conversation.
On a side note, a simple scalability paradigm that could be handy is the scalability cube, described in The Art of Scalability book. It makes you think of different directions in which your application could scale – (X) horizontal duplication, (Y) functional decomposition, and (Z) data partitioning.