Key Responsibilities:

• Design, develop, and maintain scalable and reliable ETL/ELT pipelines on Azure.
• Work closely with data scientists, analysts, and other stakeholders to deliver end-to-end data solutions.
• Build and optimize big data pipelines using PySpark and Azure Databricks.
• Orchestrate data workflows using Apache Airflow or Azure Data Factory.
• Implement and support MLOps frameworks and deployment pipelines.
• Monitor, troubleshoot, and ensure data quality and reliability across systems.
• Drive architectural decisions for data platform components, ensuring best practices are followed.
• Mentor junior data engineers and contribute to knowledge sharing within the team.

Key Skills and Technologies:

• Cloud Platforms: Microsoft Azure (Azure Data Lake, Azure Databricks, Azure Synapse, Azure Blob Storage)
• Programming Languages: Python (strong), SQL
• Big Data Technologies: PySpark, Spark
• ETL Tools: Azure Data Factory, custom ETL development
• Workflow Orchestration: Apache Airflow
• DevOps/MLOps: Azure ML, MLflow, CI/CD pipelines for ML models
• Version Control: Git, GitHub/Azure DevOps
• Other Tools (Preferred): Terraform, Docker, Kubernetes