Unleash your talents and venture into a new career that innovates and transforms products &

services globally!

Senior Data Engineer

Location: Indore

Experience: 5+ Years

[Key Responsibilities]

• Gather and assemble large, complex sets of data that meet non-functional and
  
  

functional business requirements.

Skills: SQL, Python, R, Data Modeling, Data Warehousing, AWS (S3, Athena).

• Create new data pipelines or enhance existing pipelines to accommodate
  
  

non-standard data formats from customers.

Skills: ETL Tools (e.g., Apache NiFi, Talend), Python (Pandas, PySpark), AWS Glue, JSON, XML,

YAML.

• Identify, design, and implement internal process improvements, including
  
  

re-designing infrastructure for greater scalability, optimizing data delivery, and automating manual

processes.

Skills: Apache Airflow, Terraform, Kubernetes, AWS Lambda, CI/CD pipelines, Docker.

• Build and maintain required infrastructure for optimal extraction,
  
  

transformation, and loading (ETL) of data from various data sources using AWS and SQL technologies.

Skills: SQL, AWS Redshift, AWS RDS, EMR (Elastic MapReduce), Snowflake.

• Use existing methods or develop new tools/methods to analyze the data and perform required data sanity
  
  

validations to ensure completeness and accuracy as per technical and functional requirements.

Skills: Python (NumPy, Pandas), Data Validation Tools, Tableau, Power BI.

• Work with stakeholders including Customer Onboarding, Delivery, Product,
  
  

and other functional teams, assisting them with any data-related technical or infrastructure-related issues.

Skills: Stakeholder Communication, JIRA, Agile Methodologies.

• Provide actionable insights into key data metrics (volumes, trends, outliers,
  
  

etc.), highlight any challenges/improvements, and provide recommendations and solutions to relevant

stakeholders.

Skills: Data Analysis, Data Visualization Tools (Tableau, Looker), Advanced Excel.

• Coordinate with the Technical Program Manager (TPM) to prioritize discovered
  
  

issues in the Data Sanity Report and own utility communications.

Skills: Project Management Tools, Reporting Tools, Clear Documentation Practices.

Expectations from the Role:

• Be the primary owner of the data sanity process internally
  
  

Skills: Data Quality Management, Python (Data Validation Libraries), SQL Auditing

Run defined sanity checks/validations on utility data to ensure data accuracy and completeness.

Skills: Python, SQL, QA Tools, AWS QuickSight.

Create the data sanity report using the established process and template.

Skills: Report Automation Tools, Excel Macros, Python.

Provide actionable insights into key data metrics, highlighting any challenges/improvements in the data.

Skills: Business Intelligence Tools (Tableau, Power BI), Statistical Analysis Tools.

Assess issues in the data as per the technical and functional requirements of the implementation and flag

critical issues to the TPM.

Skills: Root Cause Analysis, Data Profiling, Monitoring Tools.

Implement internal solutions to handle all issues in the data, such as creating custom transformers or

pre-ingestion logic.

Skills: Python (ETL Scripts), PySpark, AWS Lambda, JSON Processing.

Maintain and use data pipelines and processes to ingest required data in Bidgely.

Skills: AWS Data Pipeline, Apache Kafka, SQL.

In case of custom utility data formats, create custom data pipelines in Bidgely to support the ingestion

process.

Skills: Python, Apache Spark, AWS Glue.

On rare occasions, support utility data integration work by working with the utility's Datalake/IT team to

understand their data structure, data sources, and formats, and assist in extracting data from their source

systems.

Skills: Data Lake Architecture (Azure Data Lake, AWS Lake Formation), API Integration, Data

Formats (Parquet, ORC).

💼 We offer:

• Competitive salary
• Flexible work arrangements
• Professional growth opportunities
• Collaborative work culture
  
  

Powered by JazzHR

rRYztydVRm