Etl Tool To Pull Data Using Graph Sql

About this Service

The Problem

Shiphero company is an organization providing shipping solutions to vendors. The data created by shiphero for different product picking and packing time period doesn’t provide much insight into the efficiency of ship hero employees and other aspects that are needed and useful for vendors/brands to make better decisions for their business in order words the ‘key’ data is missing.

Our Solution

The solution is an effort to create the missing data by the existing data as we came to know that the ‘key’ data can be created by involving some deep methodologies and vast logical aspects linked to it. The incoming data from shiphero company is timestamp data therefore using this sequential data we can create the missing data we need to get the required KPI’s. 

The overall architecture included getting data from shiphero through api doing some preprocessing and creating our ‘key’ through this data and populating it on Google big query. This google big query is linked to Google data studio for insights visualisation.

Solution Architecture

The data coming from Shiphero is extracted every day using a cron job scheduler. Google app engine service is used to preprocess and apply a transformation to the data.

Deliverables

Ready-to-use Google data studio Dashboard. Google app engine service-based scheduler code.

Tools used

  • Google App engine

  • Google big query 

  • Google data studio

  • Google cloud platform

Language/techniques used

  • Python (for preprocessing)

  • GraphQL (For data extraction)

Skills used

  • Python Programming

  • GraphQL querying

  • Statistics

  • Data visualization

  • Data Engineering

  • Data Science

Databases used

  • Google big query

Web Cloud Servers used

  • Google Cloud platform

$600
Quick Hire
Concepts and revisions: 1 concept, 3 revisions
Project Duration: 2 weeks