Big Data Scaling & Apache Spark API for Digital Risk Management Company
Services Covered:

Industry
Information Technology
Duration
3 months
Budget
Less than 10K
Start Date
1 July 2020
Team Size
1-5
Engagement Model
Offshore
Project Capability Score (PCS)
77/100
Good
Project Capability Score (PCS) estimates how capable Polestar Analytics is of successfully delivering a project like this, based on its past experience, track record, and ability to handle similar work. Learn more about PCS
Client
Algoworks
Project Summary
A global IT outsourcing company engaged Polestar to scale its big-data processing capabilities and support mixed media (images, video, text) processing. Polestar implemented an Apache Spark-based API platform, optimized PySpark scripts and upgraded architecture and infrastructure. The engagement increased concurrent processing capacity from about 50 requests/min to over 1,000 requests/min and the client praised the team's technical breadth and collaboration.
Key Challenges
- Need for resources across different technology stacks for big data processing
- Requirement for prior experience with Apache Spark-based big data projects
- Scale up data processing both horizontally and vertically
- Process and analyze mixed media (images, video, text) to detect deviations from business rules
Project Deliverables
- API platform on Apache Spark
- Optimized PySpark scripts
- Architecture and infrastructure upgrades to support scaled processing
Project Solution
Polestar delivered an Apache Spark-based API platform integrated with the client's front-end, assisted the client's DevOps team, optimized PySpark scripts, and upgraded the architecture and infrastructure. They leveraged Hadoop stack technologies and Microsoft Azure Data Services, using Scala and Python where appropriate. The vendor presented an implementation approach which was followed during execution and staffed the engagement with a Data Engineer, Solution Architect and Project Manager.
Project Outcome
- System handles more than 1000 concurrent requests per minute (previously ~50 requests per minute)
- Optimized PySpark scripts and implemented parallel processing
- Smooth, collaborative workflow and timely deliverables
Platforms
API/Integration Platform
Cloud
Tech Stack
Apache SparkApache Hadoop
Python
Scala
Azure
Client Endorsement
Overall Review Rating
5
5 out of 5 stars
Timeliness
Cost Rating
Willing to Refer
Quality of Deliverables
Before the Polestar Solutions team came on board, our system could handle only 50 concurrent requests per minute. Now it handles more than a thousand. We were especially happy with their team's expertise across so many different fields.
Note: This endorsement is based on publicly available client feedback from external review sources.
More Projects by Polestar Analytics
Endorsed
Ongoing
E-commerce & Retail
Ongoing BI Development & Automation for New Look Vision Group
Services Covered:
+1 more
Endorsed
Completed
Advertising & Marketing
Power BI Financial Dashboards for Advertising & Big Data Company
Services Covered:
+1 more

