TerramEarth manufactures heavy equipment for the mining and agricultural industries. About 80% of their business is from mining and 20% from agriculture. They currently have over 500 dealers and service centers in 100 countries. Their mission is to build products that make their customers more productive.
Key points here are 500 dealers and service centers are spread across the world and they want to make their customer more productive.
There are 20 million TerramEarth vehicles in operation that collect 120 fields of data per second. Data is stored locally on the vehicle and can be accessed for analysis when a vehicle is serviced. The data is downloaded via a maintenance port. This same port can be used to adjust operational parameters, allowing the vehicles to be upgraded in the field with new computing modules.
Approximately 200,000 vehicles are connected to a cellular network, allowing TerramEarth to collect data directly. At a rate of 120 fields of data per second, with 22 hours of operation per day, TerramEarth collects a total of about 9 TB/day from these connected vehicles.
Key points here are TerramEarth has 20 million vehicles. Data is stored on the vehicle and is only available for analysis when the vehicle comes for servicing. Only 1% of the vehicles currently have the capability to stream real time data which produce 9 TB/day.
Our competitive advantage has always been in our manufacturing process, with our ability to build better vehicles for lower cost than our competitors. However, new products with different approaches are constantly being developed, and I’m concerned that we lack the skills to undergo the next wave of transformations in our industry. My goals are to build our skills while addressing immediate market needs through incremental innovations.
Key point here is the company wants to improve their vehicles while building new skills and reducing cost.
Existing Technical Environment
TerramEarth’s existing architecture is composed of Linux and Windows-based systems that reside in a single U.S, west coast based data center. These systems gzip CSV files from the field and upload via FTP, and place the data in their data warehouse. Because this process takes time, aggregated reports are based on data that is 3 weeks old.
With this data, TerramEarth has been able to preemptively stock replacement parts and reduce unplanned downtime of their vehicles by 60%. However, because the data is stale, some customers are without their vehicles for up to 4 weeks while they wait for replacement parts.
Key point here is that the company is working with stale data and hence have increased downtime.
Application 1: Data ingest
A custom Python application reads uploaded datafiles from a single server, writes to the data warehouse
- Windows Server 2008 R2
- 16 CPUs
- 128 GB of RAM
- 10 TB local HDD storage
Application 2: Reporting
An off the shelf application that business analysts use to run a daily report to see what equipment needs repair. Only 2 analysts of a team of 10 (5 west coast, 5 east coast) can connect to the reporting application at a time.
- Off the shelf application. License tied to number of physical CPUs
- Windows Server 2008 R2
- 16 CPUs
- 32 GB of RAM
- 500 GB HDD
- A single PostgreSQL server
- RedHat Linux
- 64 CPUs
- 128 GB of RAM
- 4x 6TB HDD in RAID 0
Key points here are TerramEarth has its infrastructure in a single location – US West Coast. The data from vehicles is uploaded as CSV files through FTP on the data warehouse. As the data is delayed TerramEarth is only able to reduce unplanned downtime of their vehicles by 60%. Some customer vehicles do not have replacement parts for over 4 weeks.
- Decrease unplanned vehicle downtime to less than 1 week
- Current bottleneck is mainly collection of data available for analytics. If the data collection can be improved i.e. more vehicles can be moved to Cellular connectivity the data is available more real time and hence the feedback loop can be completed earlier.
- Can be handled using Cloud Pub/Sub, Cloud IoT, Cloud Dataflow to capture the Cellular data and have analytics done using BigQuery and Cloud Machine Learning.
- Support the dealer network with more data on how their customers use their equipment to better position new products and services.
- can be handled using running analytics over collected data regarding the usage and consumption
- Have the ability to partner with different companies—especially with seed and fertilizer suppliers in the fast-growing agricultural business—to create compelling joint offerings for their customers.
- can be handled using building APIs to expose the data externally and using Cloud Endpoints to expose the APIs.
- Expand beyond a single datacenter to decrease latency to the American midwest and east coast
- Can be handled using multi-regional Cloud Storage and other AWS managed services like Pub/Sub, Dataflow and BigQuery
- Create a backup strategy
- data can be easily backed up in Cloud Storage or BigQuery
- Increase security of data transfer from equipment to the datacenter
- data can be transferred using HTTPs for more security
- Improve data in the data warehouse
- can be handled using BigQuery as the data warehouse solution instead of a single PostgreSQL which is limited in capability and scalability
- Use customer and equipment data to anticipate customer needs
- can be handled using running machine learning models over the data collected
Reference Cellular Upload Architecture
Batch Upload Replacement Architecture
- Google Cloud – TerramEarth case study