endorsedOngoing

Big Data Scaling & Apache Spark API for Digital Risk Management Company

Services Covered:

Big Data Analytics and Business Intelligence

Industry

Information Technology

Duration

73 months

Budget

Less than 10K

Start Date

1 July 2020

Team Size

1-5 Employees

Engagement Model

Offshore

Project Capability Score (PCS)

77/100

Good

Project Capability Score (PCS) estimates how capable Polestar Analytics is of successfully delivering a project like this, based on its past experience, track record, and ability to handle similar work. Learn more about PCS

Client

Algoworks

Sunnyvale, United States of America

Project Summary

A global IT outsourcing company engaged Polestar to scale its big-data processing capabilities and support mixed media (images, video, text) processing. Polestar implemented an Apache Spark-based API platform, optimized PySpark scripts and upgraded architecture and infrastructure. The engagement increased concurrent processing capacity from about 50 requests/min to over 1,000 requests/min and the client praised the team's technical breadth and collaboration.

Key Challenges

Need for resources across different technology stacks for big data processing
Requirement for prior experience with Apache Spark-based big data projects
Scale up data processing both horizontally and vertically
Process and analyze mixed media (images, video, text) to detect deviations from business rules

Project Deliverables

API platform on Apache Spark
Optimized PySpark scripts
Architecture and infrastructure upgrades to support scaled processing

Project Solution

Polestar delivered an Apache Spark-based API platform integrated with the client's front-end, assisted the client's DevOps team, optimized PySpark scripts, and upgraded the architecture and infrastructure. They leveraged Hadoop stack technologies and Microsoft Azure Data Services, using Scala and Python where appropriate. The vendor presented an implementation approach which was followed during execution and staffed the engagement with a Data Engineer, Solution Architect and Project Manager.

Project Outcome

System handles more than 1000 concurrent requests per minute (previously ~50 requests per minute)
Optimized PySpark scripts and implemented parallel processing
Smooth, collaborative workflow and timely deliverables

Platforms

API/Integration Platform
Cloud

Tech Stack

Apache Spark
Apache Hadoop
Python
Scala
Azure

Client Endorsement

Overall Review Rating

Timeliness

Cost Rating

Willing to Refer

Quality of Deliverables

“Before onboarding the Polestar Solutions team, the system was only able to handle 50 concurrent requests per minute. But now it can handle more than a thousand concurrent requests per minute. The company was most impressed by the team's expertise across a variety of fields.”
Ravi Jain
Director & Global Head - Salesforce & Analytics
Date: 10/15/2020