Project Description:

Migration of legacy data ingestion pipelines to a shared data management system (BDMS). The work will involve:

- Datasets will be loaded into BDMD

- New tools will be created to allow data analysts to modify the datasets

- Analysts will validate and cleanse the data which will then be delivered to AWS S3

- Data will then be enriched using technologies like Databricks, AWS Lambda & Step Functions and Jupyter Notebooks/Workflows and loaded in Delta tables.

- 5+ years of experience programming in Python.

- Solid understanding of database systems and SQL including Oracle and PostgreSQL

- A Degree in Computer Science, Engineering, Mathematics, similar field of study or equivalent work experience.

- AWS experience including services like S3, Lambda, CloudWatch

- Experience with data migrations from legacy to cloud.

- Knowledge of Spark, Databricks and IaC(Terraform/CloudFormation)

- Work directly with product owners, engineers to build solutions.

- Work with data analysts/scientists to collect insight on possible data collection efficiencies and identify automation potential for manual workflows.

- Develop infrastructure as code (IaC) and embrace DevOps principles.

- Work with AWS cloud services to implement micro services using serverless technologies.