Web Crawling & Data Ingestion Platform for Energy Trade Association
Services Covered:

Industry
Energy & Natural Resources
Duration
60 months
Budget
200K - 999K
Start Date
1 March 2013
Team Size
6-10
Engagement Model
Offshore
Project Capability Score (PCS)
74/100
Good
Project Capability Score (PCS) estimates how capable Arbisoft is of successfully delivering a project like this, based on its past experience, track record, and ability to handle similar work. Learn more about PCS
Client
Confidential
Project Summary
Arbisoft augmented the client's development team to build and maintain web crawlers and a data ingestion platform that collects and stores policy website content. They conducted QA auditing of the ingestion pipeline, provided technical leadership, and augmented staff for short-term scaling needs. The work helped the client scale their dataset and contributed materially to growth from zero to roughly $1M in annual recurring revenue.
Key Challenges
- Need for deep expertise in web crawling to extract content from hundreds of external websites
- Requirement to validate that datasets are complete and accurate before post-processing and storage
- Need for flexibility to quickly scale up the team for short-term, complicated needs
Project Deliverables
- Web crawlers for collecting PDFs and website text
- Data ingestion and validation system
- QA audits of the data ingestion pipeline
- Python and Ruby on Rails application development
- Deployment on AWS infrastructure and use of Google Cloud services
- Integration with MongoDB and Elasticsearch
Project Solution
Arbisoft developed individual web crawlers to extract PDFs and website text from numerous policy websites on daily, weekly and monthly schedules, validated and aggregated those datasets, and built the surrounding data ingestion and storage workflows. They added QA engineers to audit the ingestion system, provided project management and technical leadership, and augmented the client’s team for short-term scaling needs. The engagement used Scrapy for crawling and included development work in Python and Ruby on Rails, deployed on AWS and Google Cloud with MongoDB and Elasticsearch for storage and search.
Project Outcome
- Increased annual recurring revenue from $0 to about $1 million
- Built a dataset larger than the largest online encyclopedia by leveraging collected data
- Vendor led development of major platform components and provided strong project management
- Ability to scale additional staff quickly to meet project needs
Platforms
Web
Tech Stack
Python
Ruby on Rails
MongoDB
ElasticsearchAmazon Web Services (AWS)
Client Endorsement
Overall Review Rating
5
5 out of 5 stars
Timeliness
Cost Rating
Willing to Refer
Quality of Deliverables
Our revenue has gone from zero to $1 million, partly because of Arbisoft's work and their skill at web crawling. They've led development of key components and stayed flexible about our scalability needs. They keep building trust and have become a vital part of our development.
Note: This endorsement is based on publicly available client feedback from external review sources.
More Projects by Arbisoft
Endorsed
Ongoing
Business Consulting & Services
React Native Mobile App for Talent Management Platform
Services Covered:
+3 more
Endorsed
Ongoing
Business Consulting & Services
React Native Mobile App for JobCase (Employment Platform)
Services Covered:
+1 more

