endorsedOngoing

Real-Time Portfolio Analytics Pipeline for Wealth Solutions

Services Covered:

AI and Machine Learning • Big Data Analytics and Business Intelligence • Cloud Consulting and Services

Industry

Financial Services

Duration

79 months

Budget

50K - 199K

Start Date

1 January 2020

Team Size

1-5 Employees

Engagement Model

Onshore

Project Capability Score (PCS)

80/100

Strong

Project Capability Score (PCS) estimates how capable Dataeaze Systems is of successfully delivering a project like this, based on its past experience, track record, and ability to handle similar work. Learn more about PCS

Client

Anand Rathi Wealth Services

India

Project Summary

Dataeaze Systems implemented a real-time big data processing pipeline for a wealth solutions company, using AWS EMR, HDFS and Spark to ingest and process operational data and populate MySQL data marts for BI and analytics. The solution is in production and improved processing speed and infra efficiency, while the team ran structured Agile processes with JIRA, Slack and Zoom. The client reports significant cost and speed improvements and a smooth collaboration.

Key Challenges

Implement a big data processing pipeline using AWS EMR, Spark and MySQL
Ingest data from multiple operational data stores and aggregate/compute portfolio metrics
Provide real-time (pre-business-day) portfolio views and mark-to-market calculations across multiple portfolio dimensions

Project Deliverables

Real-time data ingestion pipeline from operational databases
AWS EMR-based processing layer with HDFS and Apache Spark
Data marts and reporting databases in MySQL for BI and analytics
Production deployment of the full solution stack

Project Solution

Dataeaze implemented a real-time data ingestion and processing pipeline using AWS EMR with underlying HDFS and Apache Spark, routing processed outputs into MySQL databases for reporting and BI. The vendor was selected based on pre-sales proposals and prior experience; the small team (1 solution architect, 2 data engineers and 1 BI developer) executed development in sprints with JIRA, Slack and Zoom and deployed the stack to production.

Project Outcome

Full solution stack running in production
Infrastructure cost reduced to approximately 1/10th by using dynamic EMR instances
Processing speed improved by more than 1000%
Updated portfolios are available to customers before the start of business day (9am IST), improving customer experience

Platforms

Cloud

Tech Stack

AWS EMR
Apache Hadoop
Apache Spark
MySQL

Client Endorsement

Overall Review Rating

4.88

Timeliness

Cost Rating

Willing to Refer

Quality of Deliverables

“While the work is ongoing, the collaboration satisfies the client. Dataeaze Systems has been processing the company's data speedily. They have a structured and smooth project management process.”
Harbinder Saini
Former Consulting CTO
Date: 11/23/2020