Sr. Data Engineer (Databricks, Spark, Kafka)

Job Description

We are looking for a highly skilled and experienced Senior Databricks Data Engineer with over 5 years of expertise in building, optimizing, and maintaining data pipelines and data integration solutions. The ideal candidate will have hands-on experience with Databricks, Python, Kafka and SQL, and should be capable of working on critical data engineering projects in a fast-paced environment.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks and related technologies.
• Implement efficient data transformation processes using Python and SQL. Integrate and manage data across cloud data platforms.
• Optimize and troubleshoot data workflows for performance and reliability. Collaborate with data scientists, analysts, and stakeholders to understand data requirements.
• Ensure data quality, governance, and security standards are met across all solutions. Work on critical data engineering tasks with supervision and high accountability.

Required Qualifications

• 5+ years of hands-on experience in data engineering.
• Strong expertise in Databricks and Apache Spark.
• Proficient in Python programming for data processing, utilizing Kafka and related technologies
• Advanced Python and SQL skills for querying and managing large datasets.
• Experience in building and maintaining data lakes and warehouses on cloud platforms. Strong analytical, problem-solving, and communication skills.
• Extensive experience with CI/CD platforms (e.g., GitLab CI, GitHub Actions, Azure Pipelines, Jenkins).
• Debug and development experience of Python
• Ability to translate business needs to technical solutions and establish buy-in with stakeholders.
• Experienced at designing, and presenting data systems for customers

About the Company:

Algonomy helps consumer businesses maximize customer value by automating decisioning across their retail business lifecycle with AI-enabled solutions for eCommerce, Marketing, Merchandising, and Supply Chain. Algonomy is a trusted partner to more than 400 leading brands, with a global presence spanning over 20 countries. Our innovations have garnered recognition from top industry analysts such as Gartner and Forrester—more at www.algonomy.com.