Location: Bangalore (Hybrid)

Experience: 8+ years

About The Role

We are seeking a highly skilled Lead Data Engineer to drive the development of scalable and robust data pipelines. The ideal candidate has extensive experience in Big Data technologies, Python, PySpark, SQL, Azure Databricks, and Hadoop. You will be responsible for designing, implementing, and optimizing large-scale data processing systems to support analytics and AI/ML workloads.

Key Responsibilities

• Lead the design and development of end-to-end data pipelines for structured and unstructured data.
• Build and optimize ETL/ELT workflows using PySpark, SQL, and Azure Databricks.
• Work with Big Data technologies such as Hadoop, Hive, and Spark to process large-scale datasets efficiently.
• Design and implement data lakes, data warehouses, and real-time streaming solutions.
• Ensure data quality, reliability, and scalability by implementing best practices in data engineering.
• Collaborate with data scientists, analysts, and software engineers to enable data-driven decision-making.
• Optimize query performance and data storage strategies for cost-effective cloud solutions.
• Implement CI/CD pipelines for data workflows and manage infrastructure on Azure.
• Mentor and guide junior data engineers, fostering a culture of technical excellence.
  
  

Required Skills & Qualifications

• 8+ years of experience in data engineering.
• Expertise in Python and PySpark for large-scale data processing.
• Strong proficiency in SQL for complex queries and data transformations.
• Hands-on experience with Azure Databricks and Hadoop ecosystem (HDFS, Hive, Spark, etc.).
• Deep understanding of data modeling, ETL frameworks, and performance optimization.
• Experience with cloud-based data solutions (Azure, AWS, or GCP), particularly Azure Data Factory and Azure Synapse.
• Familiarity with data governance, security, and compliance best practices.
• Experience leading a team and managing end-to-end data engineering projects.
  
  

Nice-to-Have Skills

• Experience with real-time data processing frameworks like Kafka or Flink.
• Knowledge of MLOps and data pipeline automation.
• Knowledge or experience with Snowflake is an added plus.
• Familiarity with Terraform or Infrastructure-as-Code (IaC) for cloud deployments.
  
  

Why Join Us?

• Lead high-impact projects in a data-driven organization.
• Work with cutting-edge Big Data and cloud technologies.
• Competitive salary, flexible work options, and career growth opportunities.
  
  

How To Apply

If you are passionate about Big Data and cloud-based data engineering, we’d love to hear from you!