Cloud Analytics Modernization on AWS for Health Data Analytics Company

The Healthtech Company required modernization of its existing system for a rapidly growing customer base. Legacy systems failed to meet growth needs due to poor scalability, security, and performance.

1. Demand for unlimited scaling possibilities

The company had previously relied on desktop-based and on-premise software for its developments and internal operations.

However, when the client sought to create a robust product that combined the features of a healthcare CRM with a Business Intelligence (BI) platform, it became clear that their traditional methods would fall short.

While the product was initially designed to manage data from just a few organizations, it was envisioned to eventually handle terabytes of data. The need for rapid scalability was a significant consideration in selecting the appropriate technical solution.

2. Utmost security for medical data

Managing both personal and medical patient data requires the maximum security, especially when safeguarding the company's proprietary information. In a cloud-based, multi-tenant architecture, where our Client offers the software to multiple distinct organizations, it's critical to ensure that the data of one organization remains isolated from others.

Failure to do so can lead to data breaches, regulatory infractions, operational challenges, and diminished trust from clients. Addressing this concern at the architectural level is of paramount importance.

3. Customized solutions for each tenant take too long

For a clear and intuitive presentation of patient data, each organization (or "tenant" from a technical standpoint) receives a set of Business Intelligence dashboards tailored to their needs. But requirements differ among tenants. For instance, one might manage data for 100 patients, while another oversees 5,000. Each client has a unique set and format of data as well.

Our client faced the challenge of efficiently crafting a unique solution for every tenant. Creating a single dashboard tailored to specific requirements could take 2 weeks to 1 month. If a client requires 20 distinct BI dashboards (such as one for diabetes analytics and another for cancer risk metrics), the cumulative period could extend from 40 to 80 weeks, or over a year.
Additionally, this approach requires a significant workforce, introducing the risk of human error — a significant concern when handling sensitive data.

Implemented a comprehensive cloud-native system on AWS that addressed the core challenges and integrated seamlessly with the company's existing infrastructure.

1. Cloud-native Development with Advanced Scalability

To address the unique challenges of the healthcare sector and create a resilient, secure product, the Client opted for AWS's cloud-native development approach, recognizing its comprehensive tools and infrastructure.

The Client engaged Belitsoft for cloud-native development to leverage our prior experience in both AWS and healthcare and ensure swift and high-quality delivery.

To address the challenge of real-time automated scaling, we embraced a serverless SaaS architecture, using AWS Lambda, a serverless computing service by AWS. This service enables precise resource scaling, either based on immediate application needs or predefined parameters. For instance, the system can automatically increase resources when CPU usage surpasses 70% and reduce them when it falls below 30%.

Such a strategy guarantees optimal resource allocation at all times. Not only does it drive cost efficiencies by eliminating payment for inactive resources during low-demand periods, but it also maintains consistent application performance during peak demands.

AWS Services and Tools deployed:

• AWS Amplify
• AWS API Gateway
• Lambda

2. Data Safety With Tenant Isolation and Enhanced Access Control

Using the multi-tenant architecture of the SaaS application, our Client offers a unique solution to various organizations.

The backbone of the app is microservices, with each designed for a specific function and accompanied by its unique database. This architecture ensures compliance, isolation, and performance tailored to each tenant. So, basically, each tenant receives its own distinct copy of the software product, safeguarding their data within the company's boundaries.

On the user side, tenants have employees using the software. When a new employee joins, the company grants them access by sending an invitation. This invitation redirects them to the software where they establish a password, granting them personalized access and feature sets.

Upon logging in, Amazon Cognito verifies the employee's identity and provides a special pass (JWT token) that dictates their permissions within the system. The Amazon API Gateway then uses this token to route the employee's requests to the appropriate microservice, ensuring smooth operation. This gateway checks the tokens using a Lambda authorizer and, combined with a Lambda layer, centrally manages these tokens for organization and efficiency.

AWS Services and Tools used:

• Amazon S3 bucket
• AWS API Gateway
• JWT
• Lambda
• AWS Cognito

3. Delivering Individual Solutions for Each Client in a Month through Data Format Standardization and Pre-built BI Dashboards

STEP 1: Automate data collection and unification for quicker deployment

Each tenant has raw data in their data sources. When a tenant puts in new data, AWS SQS (Simple Queue Service) sends a notification to a queue, like dropping a letter in a mailbox.

AWS Glue Crawlers then inspect this new data, classify it, and structure it accordingly. If there are any issues with the data "message", it's put in a special box (Dead Letter Queue) for subsequent evaluation.

To monitor this entire workflow, we employ tools like AWS CloudWatch, ensuring everything runs smoothly.

Once data is appropriately processed and cataloged, a confirmation notification is generated. Redundant or outdated data "messages" are discarded to keep things organized.

This clean data is then automatically processed, computed, and stored in Redshift, a fully managed data warehouse service, optimized to aggregate and analyze data.

AWS Services and Tools deployed:

• AWS Glue
• AWS SQS

STEP 2. Real-time Data Synchronization with a Custom-made ETL Agent

The client source data is housed in the SQL Server and undergoes frequent updates. It can be any other database like MongoDB, Oracle, etc. This database is the starting point of the ETL process.

To start the data transformation for BI dashboards, we used Apache Spark, a powerful, open-source data processing framework. It can handle large datasets, perform complex transformations, and process data at high speeds. For extensive data volumes or intricate transformations, Spark can be leveraged by the custom ETL agent for efficient execution.

A custom ETL agent created by our team manages the entire ETL process, namely:

• It extracts real-time data from the client's various databases, such as SQL Server, MongoDB, or Oracle.
• Utilizes tools, notably Apache Spark, for data transformation to achieve the necessary format or architecture.
• Transports the restructured data to AWS for subsequent storage, processing, or analysis.

Upon the Client DB server's data aggregation, a notification informs users that the data is ready for visualization on the BI dashboards.

Services and Tools:

• Apache Spark

STEP 3. Designing Pre-build Frameworks for BI Dashboard Customization

After data processing and secure storage in Redshift, it's time to bring this data to life through visualizations. This is where AWS QuickSight steps in. As AWS's premier Business Intelligence (BI) service, QuickSight transforms raw data into clear and actionable insights.

Our unique advantage lies in pre-built dashboard frameworks. Instead of starting from scratch for each tenant, our Client has a foundational structure in place. Although every tenant has distinct data sets and specific needs, the core dashboard design remains the same.

With our pre-built dashboard frameworks at its disposal, QuickSight can swiftly generate dashboards tailored to each client's nuances. Even though the foundational template is the same, the final visualization uniquely represents the tenant's specific data and needs.

Our role is ensuring precise data mapping. We're tasked with connecting to the right data source and channeling the relevant information. With this approach, the once month-long process of delivering a custom dashboard has been trimmed down to just 2 days. Thus, a full solution, boasting 20 dashboards, can now be deployed in just over a month.

AWS Services and Tools used:

• AWS Redshift
• AWS QuickSight

Our Client has received a comprehensive web platform that seamlessly integrates both BI and CRM functionalities.

This cloud-native application not only boasts the utmost scalability but is tailored for rapid multi-tenant deployments.

The security of the platform remains uncompromised, ensuring that all data and processes are protected at all times.

Incorporating cutting-edge AWS tools and services throughout the cloud-native development process highlights their essential role in delivering such a sophisticated, scalable, and secure solution.