Define and maintain the overall data architecture for the organization, including data ingestion, processing, storage, and consumption layers on Azure.
Design scalable and resilient data pipelines using Azure Data Factory (ADF) and Databricks.
Develop and implement data models and data warehousing solutions leveraging Snowflake.
Design and implement real-time and near real-time data streaming solutions using relevant technologies on Azure (e.g., Azure Event Hubs, Azure Stream Analytics, Kafka on Azure).
Establish data governance policies and ensure data quality, security, and compliance.
Evaluate and recommend new technologies and tools to enhance the data platform capabilities.
Create and maintain comprehensive architectural diagrams, data flow diagrams, and technical documentation.
Implementation & Oversight
Provide technical leadership and guidance to data engineering teams during the implementation of data solutions.
Develop and enforce coding standards, best practices, and performance optimization techniques.
Oversee the development and deployment of data pipelines and ETL/ELT processes using ADF and Databricks.
Manage and optimize the performance and scalability of the Snowflake data warehouse.
Implement and manage data streaming pipelines for real-time data processing and analytics.
Integrate and manage batch processing workflows using Control-M.
Ensure proper monitoring, alerting, and logging are implemented for all data platform components.
Collaboration & Communication
Collaborate with data scientists, analysts, and business stakeholders to understand their data requirements and deliver effective solutions.
Communicate complex technical concepts clearly and effectively to both technical and non-technical audiences.
Participate in cross-functional teams to deliver end-to-end data solutions.
Mentor and guide junior data engineers.
Continuous Improvement
Stay up-to-date with the latest trends and technologies in data engineering and cloud computing.
Identify opportunities for process improvements and automation within the data platform.
Proactively address performance bottlenecks and optimize existing data pipelines.
Technical Skill Sets
Azure Cloud Technologies: Deep understanding and hands-on experience with Azure data services, including but not limited to:
Azure Data Factory (ADF)
Azure Databricks (Spark, Delta Lake)
Azure Synapse Analytics (SQL DW, Spark Pools - preferred but not strictly required)
Azure Blob Storage
Azure Data Lake Storage (ADLS Gen2)
Azure Event Hubs
Azure Stream Analytics
Azure Functions
Azure Logic Apps
Azure DevOps (CI/CD pipelines)
Data Warehousing: Extensive experience designing and implementing data warehousing solutions, specifically with Snowflake.
Data Integration (ETL/ELT): Proven ability to design, build, and optimize complex ETL/ELT pipelines using ADF and Databricks.
Streaming Technologies: Hands-on experience with designing and implementing real-time and near real-time data streaming solutions using technologies like Azure Event Hubs, Azure Stream Analytics, or similar (e.g., Kafka).
Orchestration: Strong proficiency in using Control-M for scheduling and managing batch data processing workflows.
Data Modeling: Solid understanding of different data modeling techniques (e.g., Kimball, Inmon, Data Vault).
SQL: Expert-level proficiency in SQL for data querying, manipulation, and performance tuning.
Programming Languages: Proficiency in one or more programming languages such as Python, Scala, or Java (Python strongly preferred).
DevOps & Automation: Experience with CI/CD pipelines, infrastructure-as-code (e.g., ARM templates, Terraform), and automation of data engineering tasks.
Data Governance & Quality: Understanding of data governance principles, data quality frameworks, and data security best practices.
Technical Certifications (Preferred)
Microsoft Certified: Azure Data Engineer Associate
Microsoft Certified: Azure Solutions Architect Expert
Snowflake SnowPro Core Certification
Relevant certifications in cloud computing, data management, or big data technologies.
Qualifications
Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or a related field.
15 + years of experience in data engineering, with a significant portion focused on cloud-based data platforms.
12 + years of experience in a data architect role, defining and implementing data solutions.
Demonstrated experience leading and mentoring data engineering teams.
Excellent problem-solving, analytical, and communication skills.
Ability to work independently and as part of a collaborative team.
Strong understanding of data security and compliance requirements.
Nice-to-Have Skills
Experience with other cloud platforms (e.g., AWS, GCP).
Familiarity with data science workflows and tools.
Experience with NoSQL databases.
Knowledge of containerization technologies (e.g., Docker, Kubernetes).
How strong is your resume?
Upload your resume and get feedback from our expert to help land this job
How strong is your resume?
Upload your resume and get feedback from our expert to help land this job