endorsedCompleted

EdTech Data Platform and Spark Migration Framework

Services Covered:

Data Engineering • Cloud Consulting and Services • Staff Augmentation • Big Data Analytics and Business Intelligence

Industry

Education

Duration

22 months

Budget

50K - 199K

Start Date

1 March 2021

Team Size

1-5 Employees

Engagement Model

Onshore

Project Capability Score (PCS)

77/100

Good

Project Capability Score (PCS) estimates how capable Dataeaze Systems is of successfully delivering a project like this, based on its past experience, track record, and ability to handle similar work. Learn more about PCS

Client

Confidential

Bengaluru, India

Project Summary

Dataeaze Systems built a cloud-based data platform for an education-technology company, migrating an earlier system and developing a data consolidation layer and an Apache Spark–based data migration framework. The engagement delivered a configurable, automation-driven data migration solution and established data pipelines and a data lake. Dataeaze provided on-demand skilled resources and managed the work using Agile methodology and Jira.

Key Challenges

Build a modern data platform on AWS cloud
Migrate the legacy system to a modern cloud data platform
Implement end-to-end data engineering automation

Project Deliverables

Data consolidation layer
Spark-based data migration framework
Migration of legacy systems to a modern cloud data platform
Cloud-based data lake and data pipelines

Project Solution

Dataeaze leveraged prior experience to build a modern AWS-based data platform. The team built a data consolidation layer ingesting data from various RDBMS and NoSQL sources, migrated an earlier system into the cloud platform, and developed a configurable Spark-based data migration framework. Work was executed by 2–4 resources at different stages, including a principal data engineer, a senior data engineer, and a data engineer, and managed using Agile practices and Jira.

Project Outcome

Cloud-based data lake and data pipelines are in place and are being developed
Framework is in place for smooth configuration-driven data movement
Agile project management using Jira and access to on-demand talent

Platforms

Cloud

Tech Stack

Apache Spark
Amazon Web Services (AWS)

Client Endorsement

Overall Review Rating

4.13

Timeliness

Cost Rating

Willing to Refer

Quality of Deliverables

“Thanks to Dataeaze Systems's work, the client's cloud-based data lake and data pipelines are in place, and the framework enables smooth data movement. Dataeaze Systems provided talented resources as needed throughout the project and managed the project well using the Agile methodology and Jira.”
Kamalanathan Viswanathan
Director of Data Engineering
Date: 02/12/2023