Azure Data Engineer/ETL Engineer

Job Description: We are seeking a skilled Azure ETL Developer to join our team. The ideal candidate will have extensive experience in developing ETL pipelines, scheduling using Azure Data Factory (ADF) and/or Apache Airflow, and performance tuning using SQL. This role requires technical expertise in SQL, Python, Spark, PySpark, and Azure.

Key Responsibilities:

• ETL Pipeline Development:

1. Design, develop, and maintain ETL pipelines to extract, transform, and load data from various sources into Azure data platforms.
2. Optimize ETL processes for performance and scalability.
3. Implement data quality and validation checks within ETL processes.

• Scheduling and Orchestration:

1. Develop and manage data integration workflows using Azure Data Factory (ADF) and/or Apache Airflow.
2. Schedule and monitor ETL jobs to ensure timely data processing and availability.

• Performance Tuning:

1. Optimize SQL queries and data processing tasks for performance.
2. Utilize notebooks for development and performance tuning of ETL processes.

• Data Integration and Management:

1. Develop and maintain data models and schemas.
2. Ensure data integrity, consistency, and security across data platforms.
3. Collaborate with data engineers, solution architects, and other stakeholders to understand business requirements and translate them into technical solutions.

Qualifications:

• Bachelor's degree in Computer Science, Information Technology, or a related field.
• 5-6 years of experience in developing ETL pipelines and scheduling using ADF and/or Apache Airflow.
• Strong technical skills in SQL, Python, Spark, PySpark, and Azure.
• Proven experience in performance tuning and optimization of ETL processes.
• Experience in healthcare and pharmaceutical industries is preferred.
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration skills.

Preferred Skills:

• Experience with other cloud platforms (AWS, Google Cloud).
• Knowledge of big data technologies (Hadoop, Kafka).
• Familiarity with data visualization tools (Power BI, Tableau).