We seek a highly skilled and experienced Data Engineering Lead to join our team. This role demands deep technical expertise in Apache Spark, Hive, Trino (formerly Presto), Python, AWS Glue, and the broader AWS ecosystem. The ideal candidate will possess strong hands-on skills and the ability to design and implement scalable data solutions, optimize performance, and lead a high-performing team to deliver data-driven insights.

Technical Leadership

The core responsibilities for the job include the following:

• Lead and mentor a team of data engineers, fostering best practices in coding, design, and delivery.
• Drive the adoption of modern data engineering frameworks, tools, and methodologies to ensure high-quality and scalable solutions.
• Translate complex business requirements into effective data pipelines, architectures, and workflows.
  
  

Data Pipeline Development

• Architect, develop, and optimize scalable ETL/ELT pipelines using Apache Spark, Hive, AWS Glue, and Trino.
• Handle complex data workflows across structured and unstructured data sources, ensuring performance and cost-efficiency.
• Develop real-time and batch processing systems to support business intelligence, analytics, and machine learning applications.
  
  

Cloud And Infrastructure Management

• Build and maintain cloud-based data solutions using AWS services like S3 Athena, Redshift, EMR, DynamoDB, and Lambda.
• Design and implement federated query capabilities using Trino for diverse data sources.
• Manage Hive Metastore for schema and metadata management in data lakes.
  
  

Performance Optimization

• Optimize Apache Spark jobs and Hive queries for performance, ensuring efficient resource utilization and minimal latency.
• Implement caching and indexing strategies to accelerate query performance in Trino.
• Continuously monitor and improve system performance through diagnostics and tuning.
  
  

Collaboration And Stakeholder Engagement

• Work closely with data scientists, analysts, and business teams to understand requirements and deliver actionable insights.
• Ensure that data infrastructure aligns with organizational goals and compliance standards.
  
  

Data Governance And Quality

• Establish and enforce data quality standards, governance practices, and monitoring processes.
• Ensure data security, privacy, and compliance with regulatory frameworks.
  
  

Innovation And Continuous Learning

• Stay ahead of industry trends, emerging technologies, and best practices in data engineering.
• Proactively identify and implement improvements in data architecture and processes.
  
  

Requirements

• Advanced proficiency with Apache Spark (core, SQL, streaming) for large-scale data processing.
• Strong expertise in Hive for querying and managing structured data in data lakes.
• In-depth knowledge of Trino (Presto) for federated querying and high-performance SQL execution.
• Solid programming skills in Python with frameworks like PySpark and Pandas.
• Hands-on experience with AWS Glue, including Glue ETL jobs, Glue Data Catalog, and Glue Crawlers.
• Deep understanding of data formats such as Parquet, ORC, Avro, and their use cases.
• 8+ years of experience in building data pipelines from scratch in large data volume environments
• AWS certifications, such as AWS Certified Data Analytics or AWS Certified Solutions Architect.
• Experience with Kafka or Kinesis for real-time data streaming would be a plus.
• Familiarity with containerization tools like Docker and orchestration platforms like Kubernetes.
• Knowledge of CI/CD pipelines and DevOps practices for data engineering.
• Prior experience with data lake architectures and integrating ML workflows.
• Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field.
• 8+ years of experience in data engineering with a minimum of 2 years in a leadership role.
  
  

Cloud Proficiency

• Expertise in AWS services, including S3 Redshift, Athena, EMR, DynamoDB, and IAM.
• Experience designing scalable and cost-efficient cloud-based data solutions.
  
  

Performance Tuning

• Strong ability to optimize Apache Spark jobs, Hive queries, and Trino workloads for distributed environments.
• Experience with advanced techniques like partitioning, bucketing, and query plan optimization.
  
  

Leadership And Collaboration

• Proven experience leading and mentoring data engineering teams.
• Strong communication skills, with the ability to interact with technical and non-technical stakeholders effectively.
  
  

This job was posted by Pavithra Nagamani from Innovatily.