endorsedCompleted

Cloud Dev & SI for Data Transformation Platform

Services Covered:

Cloud Consulting and Services

Industry

Information Technology

Duration

6 months

Budget

50K - 199K

Start Date

1 September 2019

Team Size

1-5 Employees

Engagement Model

Offshore

Project Capability Score (PCS)

77/100

Good

Project Capability Score (PCS) estimates how capable Velotio Technologies is of successfully delivering a project like this, based on its past experience, track record, and ability to handle similar work. Learn more about PCS

Client

Confidential

Sunnyvale, United States of America

Project Summary

Velotio Technologies ported the client's infrastructure to the AWS cloud and built a cloud-based ingestion solution to drop, access, and transform customer data. The ingestion pipeline (built with S3, Athena, and Python) ingests multiple gigabytes per hour with monitoring and alerts; the DevOps/cloud migration work had more limited outcomes. Communication during the ingestion project was strong and stakeholders were satisfied with the data pipeline's accuracy and throughput.

Key Challenges

Migrate infrastructure to the Cloud (DevOps)
Build a robust cloud-based ingestion solution for customer data

Project Deliverables

Infrastructure ported to the AWS cloud
A cloud-based ingestion pipeline using S3, Athena, and Python
Monitoring and alerting for ingestion and transformation jobs
A data pipeline designed to avoid data loss and enable queryable outputs

Project Solution

Velotio ported the client's infrastructure to AWS using tools such as Ansible, Docker, and Kubernetes and worked across AWS and Google Cloud to enable deployment across clouds or customer infrastructure. For ingestion, the team implemented a pipeline using S3, Athena, and Python that allows customers to drop data into the cloud, reshapes data into optimized forms via queries, provides monitoring and alerting, and is designed to prevent data loss while enabling queries of transformed outputs.

Project Outcome

Built a pipeline able to ingest multiple gigabytes of data per hour with accurate monitoring and alerts
Met product accuracy and ingestion rate expectations across processed sites
DevOps/cloud deployment results were muted and some deployment metrics were not fully achieved

Platforms

Cloud

Tech Stack

Amazon Web Services (AWS)
Python

Client Endorsement

Overall Review Rating

4.13

Timeliness

Cost Rating

Willing to Refer

Quality of Deliverables

“To stakeholder satisfaction, Velotio Technologies delivered an ingestion solution that consumed a significant amount of data precisely and efficiently. During the solution’s development, the team effectively communicated to mitigate oversight. They offered diverse expertise, which was valuable.”
Anonymous
Lead Engineer
Date: 07/07/2020