Job Description

• 12+ years of experience in data engineering with Databricks and Spark
• Strong knowledge of Databricks architecture, tools and capabilities
• Experience in Design, develop, and optimize ETL/ELT data pipelines using Databricks on AWS.
• Good understanding of AWS native services – S3, DataSync, DMS, Lambda, SNS, SQS.
• Deep understanding of distributed computing and how to use spark for data processing
• SQL and PySpark : strong command over querying databases and proficiency in PySpark.
• Experience with Delta Lake and building data lakes.
• Familiarity with CI/CD pipelines using tools such as Gitlab, or CodePipeline.
• Experience with infrastructure as code (IaC) tools like Terraform or AWS CloudFormation.
• Strong analytical and problem-solving skills, with the ability to optimize complex data workflows.
• Excellent communication skills and the ability to work in a fast-paced, collaborative environment
• Experienced in complex customer interactions
• Ability to drive technical and business conversations with C-level executives of financial institutions to develop relationships, orchestrate teams and influence outcomes to achieve long-term customer success.
• Skilled in identifying and prioritizing key use cases, resource allocations, and investment focus.
• Prior experience with coding in a core programming language (i.e., Python, Java, Scala) and willingness to learn a base level of Spark.
• Hands-on expertise with complex proofs-of-concept and public cloud platform(s).
• Know how to provide technical solutions for specialized customer needs and navigate a competitive landscape.

Roles & Responsibilities:

• Accountable for the design, development and architecture of new and current systems
• Participates in enterprise architecture reviews and provides recommendations and guidance to
• align with corporate systems
• Act as technical lead for medium to large complex project/feature enhancements
• Work with the Product Management team, Project Management, Business and User Experience teams
• to identify and prioritize initiatives and features
• Participate in all grooming and priority setting sessions and accountable for project sizing and estimation
• Evaluates new tools, technologies and processes to improve speed, efficiency and scalability of Datalake Platform
• Work closely with other development teams across organization to troubleshoot issues, performance
• tuning and identify opportunities for seamless integrations across systems
• Works closely with the Production Support team to train, guide and troubleshoot production issues
• Provides coaching and technical guidance for Software Engineers at different levels of experience
• Thrives on working in a fast-paced environment.

#LI-RJ2

Salary Range-$100,000-$150,000 a year