Role: Databricks Architect

Location: Minneapolis, MN (Onsite- 5 days )

Job Description:

• Candidate to build a Delta Lake. An understanding on Medallion architecture is important.
• Designed Batch and Real time data ingestion end to end.
• Data Lineage using Data Catalog – maintained that and used the tool to discover data sources and targets;
• Designed Data Quality Strategies – identify CDE -critical data elements, assigning data stewards from business side and coded the DQ metrics generation in curated zone/silver layer.
• Designed CDC/incremental loads, full load strategies – using hash keys, key columns and merge designs.
• Designed Data Security – PII/NPI, RBAC- role based access control.
• Azure Data factory pipeline creation Azure DevOps
• Designed using Azure Data factory pipeline and Databricks – writing out to azure blob and then finally to snowflake as consumption were from Power BI /snowflake.
• Initially worked with client team and did extensive data analysis to create the model, data delivery mechanism, cloud storage, retention requirement, security, RBAC [role based access control]…etc
• Streaming datasets/file- were captured using AUTOLOADER – delta live tables into raw and designed to be updated into silver, gold.
• Provide guidance on Azure cost management strategies, including resource tagging, budgeting, and monitoring.
• Conduct Azure security assessments, vulnerability scanning, and threat modeling to enhance cloud security.
• Discussing potential projects and their parameters with clients, executives, and software developers.
• Planning out the blueprints for software projects, including defining the scope, allocating resources, setting deadlines, laying out communication strategies, and indicating tests and maintenance.
• Assembling and leading the project team.
• Participating in and supervising each stage of the project.
• Ensuring each project stays on schedule and adheres to the deadlines.
• Determining and overseeing consistent testing, evaluation, and troubleshooting of all products in all stages of completion.
• Tracking milestones, deliverables, and change requests.
• Serving as a liaison to communicate information regarding changes, milestones reached, and other pertinent information.
• Preparing project proposals and discussing potential projects with clients and stakeholders
• Facilitating project initiation by defining project scope and requirements, and preparing the necessary documents and requirements
• Developing project plans and timelines to ensure the timely submission of project deliverables
• Managing project budgets and resources to ensure the timely completion of milestones
• Identifying and managing project risks
• Facilitating team meetings and collaboration
• Liaising for changes and negotiations with relevant stakeholders
• Ensuring software quality standards are met and requirements are submitted within budget and on time
• Closing the project and ensuring proper documentation"