Our company developed a payments orchestration platform. 

In this article, we shared our experience of moving to cloud microservice architecture and pointed out the practical benefits of this approach. This guide should be of great interest to junior DevOps looking for a basic understanding of complex stand-alone cloud-based systems and seasoned Fintech experts facing the limitations of their current solutions' architecture.

Being among the pioneers of the e-payment processing solutions, in 2012, we built the platform architecture based on proven and popular at the time solutions - PCI DSS ready infrastructure with hardware firewalls Cisco ASA, network segmentation. We also used separate hosting services for every role - Front end with an admin panel, payment forms, API and incoming/outgoing proxy, processing; hosting services for issuing vacant keys for card data encryption; hostings with relational databases, etc. The tech stack was pretty basic, too - PHP, Apache, MySQL.

<PROXIMA SCHEME>

For a long time, the platform served its purpose perfectly. While processing around 5 million transactions a month from several large worldwide clients, it proved to be an effective tool with a high level of fault tolerance and excellent speed.

However, with time, we noticed a range of issues that hindered the system's performance.

For instance, the relational databases reached extreme volumes, and our partitioning attempts only solved the speed issue for a short time. Later, we decided to split big tables. Yet, fetching data from slaves remained the bottleneck when displaying statistics in the admin panel.

To implement horizontal scaling, we ordered new nods - and this led to the shutdown of the entire availability zone throughout the maintenance (availability zones were geographically distributed and identical in composition and configuration of servers / Cisco).

Among other problems we stumbled upon was the legacy code and specific solutions on the development and admin sides.

Platform maintenance became a real issue. And as a result, the company's sales volumes dropped: selling the product of this quality was beyond difficult. After a couple of attempts of deep refactoring, we concluded that we had to fundamentally change the platform and implement new technologies that had already become ubiquitous - Docker, Kubernetes, ELK.

With the support of the management, we formed a core action group responsible for building the project from scratch.

We developed an MVP that, besides the processing core, also included:

• An admin panel with basic functionality.

• Several connectors to acquiring banks.

• Scripts for old platform data migration.

As soon as we launched the MVP and conducted a series of tests and system demos, we started looking for solutions that would allow us to faster "get the feel" of the platform. So, naturally, we considered ready-made Amazon AWS cloud solutions - EC2, RDS, Elastic, Load balancer, etc. This would help us launch fast and with minimum involvement of the DevOps. But the price was too steep. With multiple Availability Zones within AWS itself, disc encryption, and Elastic, it all added up to USD 2000/month on the test stage, and the cost of the solution in production would skyrocket.

We knew that we could also continue in-house system optimization with little to no assistance from AWS box solutions. But that would translate into the loss of 99.99% for the whole set.

Long story short, after close consideration, we decided to go with the second option - and started building our own cloud infrastructure on dedicated servers, which (spoiler alert!) had proven time and again to be effective.

<RAFINITA SCHEME>

In the new system, hardware firewalls are the system core. The only difference from the previous scheme is in Juniper SRX models chosen over the Cisco ASA ones we used before.

There are several reasons behind this move. First of all, they are more popular among hosting service providers. Second of all, they support stateful filtering and work across levels on the OSI model.

Fault tolerance is ensured by several availability zones working on the active-standby principle. For it, we utilized a simple automatic balancing on the DNS level of the Cloudflare service. Each zone is now a Proxmox VE cluster ensuring both fault tolerance within the zone and convenience in managing all the virtualization hostings with a single console.

We implemented the "one server - one feature" principle on the microservice level. So, incoming traffic passes through the Juniper firewall, NAT, NGINX Proxy with Web application firewall (WAF), reaches Kubernetes LoadBalancer, and is then redirected to one of the microservices. The latter work with three services outside the Kubernetes cluster: RabbitMQ, Elasticsearch, Percona XtraDB (MySQL). For asynchronous master-master replication of the transactional databases between availability zones, we utilized Percona replication manager, which, in its turn, is based on an encrypted IPSec tunnel through Juniper.

We also implemented changes in the development/testing: besides transitioning to the Symfony frameworks, Vue.js, TypeScript for the frontend, git, and transparent CI / CD processes, we have significantly improved the quality of the code by switching to the SOLID approach in design as well as the use of automatic checks like phpmd, phpcpd, phpcs, phpstan, eslint, etc. In the meantime, QA is mainly using Behat and Selenium.

We're sure that building a payments orchestration platform from scratch was a great move both for the system itself and the development team behind it. Moreover, it has completely reshaped the product concept. From a semi-closed platform focused on processing a limited sales volume for just a few large clients, the system took a leap forward, becoming a unique ready-made solution fully ready for B2B promotion in a market with fierce competition.