Responsibilities :

Data Pipeline Development & Management :

• Design, build, and maintain highly scalable and efficient data pipelines on Google Cloud Platform (GCP).
• Develop and implement ETL/ELT processes to ingest, transform, and load large datasets.
• Optimize data workflows for performance, reliability, and cost-effectiveness.
  
  

GCP Service Utilization :

• Leverage and optimize Google Cloud Platform services, including BigQuery, Dataflow, Pub/Sub, Cloud Storage, and Cloud Functions.
• Utilize Dataflow for stream and batch data processing.
• Implement Pub/Sub for real-time data ingestion and messaging.
• Design and manage data warehouses within BigQuery.
  
  

Data Integrity & Security :

• Ensure data integrity, quality, and consistency across all data pipelines.
• Implement and maintain data security measures, adhering to best practices and compliance requirements.
• Monitor data pipelines for errors and anomalies, and implement proactive measures to prevent issues.
  
  

Collaboration & Communication :

• Collaborate effectively with cross-functional teams, including data scientists, analysts, and software engineers, in an agile environment.
• Communicate technical concepts clearly and concisely to both technical and non-technical stakeholders.
• Participate in code reviews and contribute to the development of best practices.
  
  

Troubleshooting & Problem Solving :

• Troubleshoot and resolve complex data-related issues, ensuring minimal downtime and disruption.
• Perform root cause analysis and implement long-term solutions to prevent recurring issues.
  
  

Experience :

Required Skills and Experience :

• 4+ years of hands-on experience as a Data Engineer, with a strong emphasis on Google Cloud Platform (GCP).
  
  

GCP Expertise :

• Extensive experience with core GCP services, including:
• BigQuery (data warehousing and analytics)
• Dataflow (stream and batch data processing)
• Pub/Sub (real-time messaging)
• Cloud Storage.
• Cloud Functions.
• Strong understanding of GCP data architecture and best practices.
  
  

Programming & Scripting :

• Proficiency in Python for data processing and automation.
• Strong SQL skills for data querying and manipulation.
  
  

Infrastructure & Automation :

• Knowledge of Infrastructure as Code (IaC) using Terraform (preferred).
• Experience with CI/CD pipelines for automated deployments.
  
  

Problem Solving & Communication :

• Strong problem-solving and analytical skills.
• Excellent communication and collaboration skills.
• Ability to work Qualifications :
  
  

Experience :

• 5+ years of experience in GCP Data Engineering.
• Experience in optimizing data workflows and implementing complex ETL/ELT processes.
  
  

Advanced Skills :

• Experience with DataProc.
• Experience with Airflow or Cloud Composer.
• Experience with Data Catalog.
  
  

(ref:hirist.tech)