S P A Enterprise Services - Data Engineer - Python/Spark
Hyderabad, TS, IN
2 months ago
Save Job
Summary
Job Description
Responsibilities :
Design, develop, and maintain efficient and reliable data pipelines using PySpark, Python, and AWS services (S3, EC2, Lambda, Redshift, CloudFormation, etc.).
Implement data transformation logic, data cleansing, and data enrichment processes to ensure data quality and consistency.
Optimize data pipelines and queries using Spark SQL and PySpark to improve performance and scalability.
Refactor and modernize legacy codebases, focusing on clean code, improved readability, and maintainability.
Utilize CloudFormation to provision and manage AWS infrastructure in an automated and repeatable manner.
Implement and adhere to best practices for code versioning using Git and artifact management using JFrog Artifactory.
Develop and execute comprehensive unit tests and implement
Test-Driven Development (TDD) methodologies to ensure code quality and reliability.
Diagnose and resolve complex bugs and performance issues in data pipelines and applications.
Collaborate effectively with data scientists, analysts, and other engineers to understand data requirements and deliver solutions.
Maintain a strong understanding of AWS architecture and best practices, and apply them to design and implement efficient and scalable data solutions.
(ref:hirist.tech)
How strong is your resume?
Upload your resume and get feedback from our expert to help land this job
How strong is your resume?
Upload your resume and get feedback from our expert to help land this job