Lead Data Engineer – DWH Cloud

As a Lead Data Engineer – DWH for Cloud, you will be leading DWH projects and responsible for designing end to end technical solutions on AWS for various reporting, integrations and analytics related business needs along with team management, timely delivery and ensuring complete support on delivered pipelines.

Lead Data Engineer will typically have:

• 9+ years of experience into Data field
• Experience in designing and architecting data solutions that support organization’s strategic business goals.
• End to end ETL and ELT processes understanding for Data ingestion and transformation in AWS, and coordination amongst the team along with reporting and visualizations.
• Should be able to design data models, performance optimization processes for data product
• Should be able to monitor, manage and ensure best practices to support data platform
• Should be well versed with extraction, transformation, loading, data cleansing, data quality, data visualization, data monitoring concepts to instill capabilities in the team for building right processes for the data platform
• Should ensure reusability, cost optimization and better performance of the processes being followed on data platform.
• Build, maintain, and monitor, batch and real-time ETL pipelines in an AWS architecture (Kinesis, DMS, S3, EMR, RedShift, etc.)
• Carries basic understanding of Golden Gate.
• Work closely with the Data Analytics teams to develop a clear understanding of data and data infrastructure needs; assist with data-related technical issues
• Develop Data strategy (source, flow of data, storage, and usage)
• Perform Data validation and quality assurance
• Present technical solutions to various stakeholders
• Provide day-to-day support of the DW and DL environments, with excellent communications across teams, monitor new deployments and services, escalating issues where appropriate.

Who we prefer?

• Data Warehousing concepts (min 5 years)
• Hands on experience on AWS (min 3 Years)
• Building ETL pipelines (min 5 years)
• Job Scheduling Frameworks (min 3 years)
• Performance tuning of SQL queries (min 3 years)
• Knowledge of Spark – Data frames, Spark-Shell, Lazy evaluation, performance understanding
• Knowledge/experience of SQL - should be very conversant on Spark SQL
• Python skills (min 2 years)
• Visualization skills (min 1 year)
• Data modeling, architecture, and Design data systems (min 3 years)
• Experience in handling Client/business teams
• Documentation Skills

Education Qualification: BE / B.Tech / M.Tech