As a Senior Data Engineer, you will be responsible for designing, developing, and optimizing data infrastructure and pipelines, ensuring data integrity, and enabling analytics and machine learning initiatives. You will work closely with data scientists, analysts, and software engineers to architect and implement data solutions that drive business insights and operational efficiency. Knowledge of Kubernetes and containerized environments is essential for managing workloads and scaling data solutions effectively.

Key Responsibilities

• Design, build, and maintain scalable and efficient data pipelines using modern data engineering tools and frameworks.
• Develop ETL/ELT processes to ingest, transform, and store structured and unstructured data from various sources.
• Implement and manage data orchestration frameworks to automate, schedule, and monitor data workflows efficiently.
• Optimize data storage, retrieval, and processing performance for large-scale datasets.
• Design and implement data warehouses and data lakes to support analytics, reporting, and machine learning workloads.
• Manage and maintain on-premises and cloud data infrastructure, including storage, networking, and compute resources.
• Deploy and manage containerized data workloads using Kubernetes.
• Collaborate with cross-functional teams to understand business needs and provide data-driven solutions.
• Develop and implement dashboards to visualise data.
• Ensure data quality, security, and governance best practices are followed.
• Implement and maintain data lineage tracking to ensure traceability and transparency of data transformations.
• Implement monitoring and alerting systems for data pipelines to ensure reliability and uptime.
• Drive the adoption of best practices in data engineering, automation, and on-premises infrastructure management.
• Lead initiatives and mentor junior engineers and contribute to a culture of continuous learning and improvement.
  
  

Requirements

• Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field.
• 5+ years of experience in data engineering, data architecture, or software development with a focus on data solutions.
• Strong experience in on-premises data infrastructure, including storage, networking, and compute resources.
• Proficiency in programming languages such as Python and Java.
• Hands-on experience with Kubernetes for container orchestration and workload management.
• Strong experience with big data processing frameworks such as Apache Spark, or similar.
• Experience designing and implementing relational database, object storage, data warehouses and data lakes (e.g. MariaDB, MinIO, AWS S3, ClickHouse, etc.).
• Familiarity with data orchestration tools like Apache Airflow or similar.
• Experience in data modelling, warehousing, and analytics solutions.
• Knowledge of data visualization tools such as Grafana, Apache Superset, or similar for presenting insights effectively.
• Strong understanding of data governance, security, and compliance best practices.
• Knowledge and experience with cloud platforms such as AWS, Azure, or GCP is beneficial.
• Experience with real-time data streaming technologies such as Kafka or Apache Flink is beneficial.
• Excellent problem-solving skills, communication, and the ability to work in a collaborative environment.