Role Description

• JOB DESCRIPTION Lead Data Engineer Job Description:
  
  

We are looking for a highly skilled Azure Data Engineer to join our growing data team. The ideal candidate will have extensive experience working with data platforms, analytics, and cloud-based solutions, particularly within the Microsoft Azure ecosystem. As an Azure Data Engineer Lead, you will be responsible for designing, building, and managing scalable data pipelines, ensuring data is properly ingested, transformed, and made available for business intelligence and analytics.

Responsibilities/Tasks

• Design, develop, and implement end-to-end data pipelines on the Azure platform using services like Azure Data Factory, Azure Databricks, Azure SQL Data Warehouse, and Azure Synapse Analytics
• Work with Lead data engineer, business analysts and SMEs to ensure the availability of high-quality data for analytics, machine learning models, and business intelligence (BI).
• Integrate various data sources (structured, unstructured, and semi-structured) into the data warehouse or data lake for reporting and analysis
• Implement and manage ETL processes using Azure Data Factory or other Azure services to move, transform, and load data from various sources.
• Optimize data storage solutions, including relational, NoSQL, and file-based storage (Azure Blob Storage, Azure Data Lake).
• Monitor data pipelines and workflows, ensuring the accuracy, reliability, and timeliness of data
• Perform data cleansing and transformation tasks to ensure data is in a usable format for analysis.
• Implement data governance best practices and ensure data security and compliance within Azure.
• Collaborate with cross-functional teams to understand business requirements and translate them into data engineering solutions.
• Stay current with the latest Azure tools and technologies, continuously improving the data engineering processes
• Work with product owners to understand business reporting requirements and deliver appropriate insights on regular basis
• Responsible for system configuration to deliver reports, data visualizations, and other solution components
  
  

Skills Required

• 5+ years of experience in Data Engineering, with minimum 3+ years of strong experience in Microsoft Azure cloud services
• Hands-on experience with Azure Data Factory, Azure Databricks, Azure SQL Data Warehouse, Azure Synapse Analytics, and Azure Blob Storage
• Proficiency in programming languages such as Python, SQL, or Scala for data engineering tasks
• Experience with data modelling, data integration, and ETL processes.
• Strong knowledge of data pipeline orchestration and scheduling in Azure.
• Familiarity with data lakes, data warehouses, and other data storage solutions.
• Knowledge of best practices for data governance, security, and privacy in a cloud environment.
• Experience working with structured and unstructured data, as well as real-time data processing.
• Strong problem-solving skills and the ability to work in a collaborative environment.
• Excellent communication skills, with the ability to work with both technical and non-technical teams
• Experience in Azure SQL Server, Azure Synapse and Azure Synapse Analytics
• Understanding of Reporting and Data Visualization tools (PowerBI or SSRS)
• Experience in migrating systems from SSIS to Azure based systems, is a plus
• Experience in the healthcare insurance industry, particularly with payer data, is a plus
• Experience in data cleansing, data engineering, data enrichment, data warehousing/ Business Intelligence preferred.
• Certification in Microsoft Azure Data Engineering or related fields is a plus.
• Familiarity with CI/CD pipelines, DevOps practices, and automation in the context of data engineering.
  
  

Knowledge of Agile, SCRUM and/or Agile methodologies. .

Skills

Azure Data Factory,Sql,Warehouse,Databricks