Agilisium Consulting

Data Engineer - AWS & Python

2 months ago

Save Job

Key Responsibilities

Design, develop, and optimize data pipelines and ETL workflows using AWS Glue, AWS Lambda, and Apache Spark.
Implement big data processing solutions leveraging AWS EMR and AWS Redshift.
Develop and maintain data lakes and data warehouses on AWS (S3, Redshift, RDS).
Ensure data quality, integrity, and governance using AWS Glue Data Catalog and AWS Lake Formation.
Optimize data storage and processing for performance and cost efficiency.
Work with structured, semi-structured, and unstructured data across various storage formats (Parquet, Avro, JSON, etc.).
Automate and orchestrate data workflows using AWS Step Functions and Apache Airflow.
Implement best practices for CI/CD pipelines in data engineering with AWS CodePipeline and AWS CodeBuild.
Monitor, troubleshoot, and optimize data pipeline performance and scalability.
Collaborate with cross-functional teams including data scientists, analysts, and software engineers.

Required Skills & Qualifications

6+ years of experience in data engineering and big data processing.
Strong experience with AWS cloud services: AWS Glue, AWS Lambda, AWS Redshift, AWS EMR, S3.
Proficiency in Python for data engineering tasks.
Hands-on experience with Apache Spark and SQL for data processing.
Experience with data modeling, schema design, and performance tuning.
Understanding of AWS Lake Formation and Lakehouse principles.
Experience with version control (Git) and CI/CD pipelines.
Knowledge of data security, compliance, and governance best practices.
Experience with real-time streaming technologies (Kafka, Kinesis) is a plus.
Strong problem-solving, analytical, and communication skills.