As a Data Engineering Manager, you will lead the design, development, and optimization of MobiKwik’s data infrastructure and pipelines. You’ll be responsible for managing a team of data engineers while working hands-on with modern Big Data and cloud-native technologies to build scalable, real-time, and robust data systems that power our products and analytics.
Job Location - Gurgaon - WFO
Key Responsibilities
● Lead the architecture, development, and deployment of large-scale data processing systems and data lakes.
● Design and implement real-time and batch data pipelines using PySpark, Spark Structured Streaming, and Big Data technologies.
● Drive the architecture of domain-centric Big Data solutions aligned with product and analytics needs.
● Provide technology leadership and make key decisions around Big Data tools, frameworks, and architecture.
● Collaborate with cross-functional teams (Data Science, Product, Engineering) to deliver data solutions. ● Own infrastructure setup, performance tuning, cost optimization, and monitoring of Spark workloads and distributed systems.
● Manage and mentor a team of high-performing data engineers.
● Ensure data quality, governance, security, and compliance across platforms.
Mandatory Skills & Experience
● 8+ years of overall experience in data engineering, with 4+ years in PySpark and Big Data tech stack.
● Strong hands-on experience with Spark Structured Streaming and large-scale stream processing.
● Proficient in Python and/or Scala for Big Data engineering.
● Solid understanding of data modeling, data architecture, and database design.
● Experience with AWS Data Platform Stack: ○ Glue, Hudi, Athena, S3, EMR, Airflow ○ Hive, MapReduce, Databricks, Snowflake
● Proven experience in cloud deployment (AWS preferred; Azure/GCP a plus).
● Deployed at least 4+ large-scale big data products, including Business Data Lakes, NoSQL DBs.
● Deep knowledge of Big Data ecosystem: Spark, Hive, Hudi, Airflow, Athena, etc.
● Comfortable with administration, performance tuning, and monitoring of Spark/Distributed systems.
● Familiarity with modern orchestration tools, preferably Airflow.
● Experience in streaming on PySpark, analytics on Big Data, and data storage frameworks. Preferred
Qualifications
● Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
● Strong leadership, communication, and stakeholder management skills.
● Hands-on problem solver with a passion for innovation and continuous learning.
● Ability to operate in a fast-paced and evolving environment.