endorsedOngoing

Web Crawling & Data Ingestion Platform for Energy Trade Association

Services Covered:

Custom Software Development • Web Scraping Services • Software QA and Testing

Industry

Energy & Natural Resources

Duration

161 months

Budget

200K - 999K

Start Date

1 March 2013

Team Size

6-10 Employees

Engagement Model

Onshore

Project Capability Score (PCS)

74/100

Good

Project Capability Score (PCS) estimates how capable Arbisoft is of successfully delivering a project like this, based on its past experience, track record, and ability to handle similar work. Learn more about PCS

Client

Confidential

Boston, United States of America

Project Summary

Arbisoft augmented the client's development team to build and maintain web crawlers and a data ingestion platform that collects and stores policy website content. They conducted QA auditing of the ingestion pipeline, provided technical leadership, and augmented staff for short-term scaling needs. The work helped the client scale their dataset and contributed materially to growth from zero to roughly $1M in annual recurring revenue.

Key Challenges

Need for deep expertise in web crawling to extract content from hundreds of external websites
Requirement to validate that datasets are complete and accurate before post-processing and storage
Need for flexibility to quickly scale up the team for short-term, complicated needs

Project Deliverables

Web crawlers for collecting PDFs and website text
Data ingestion and validation system
QA audits of the data ingestion pipeline
Python and Ruby on Rails application development
Deployment on AWS infrastructure and use of Google Cloud services
Integration with MongoDB and Elasticsearch

Project Solution

Arbisoft developed individual web crawlers to extract PDFs and website text from numerous policy websites on daily, weekly and monthly schedules, validated and aggregated those datasets, and built the surrounding data ingestion and storage workflows. They added QA engineers to audit the ingestion system, provided project management and technical leadership, and augmented the client’s team for short-term scaling needs. The engagement used Scrapy for crawling and included development work in Python and Ruby on Rails, deployed on AWS and Google Cloud with MongoDB and Elasticsearch for storage and search.

Project Outcome

Increased annual recurring revenue from $0 to about $1 million
Built a dataset larger than the largest online encyclopedia by leveraging collected data
Vendor led development of major platform components and provided strong project management
Ability to scale additional staff quickly to meet project needs

Platforms

Tech Stack

Python
Ruby on Rails
MongoDB
Elasticsearch
Amazon Web Services (AWS)

Client Endorsement

Overall Review Rating

Timeliness

Cost Rating

Willing to Refer

Quality of Deliverables

“Revenue has jumped from zero to $1 million in part due to Arbisoft’s contributions and web crawling proficiency. Leading development of critical components, their team has remained flexible to accommodate needs around scalability. They continue to foster trust as an essential development asset.”
Anonymous
VP of Engineering & Product Dev
Date: 03/21/2018