Overview
Delivery and solution leadership role that balances hands-on architectural guidance with execution ownership, client leadership, and people development. Architects the data platform blueprint and implements the design, encompassing the relevant data platform components.
What you'll do
- Architect the data platform blueprint and implement the design across the relevant data platform components.
- Collaborate with Integration Architects and Data Architects to ensure cohesive integration between systems and data models.
- Design and build complex pipelines using Delta Lake, Auto Loader, Delta Live Tables (DLT), and deployment using Asset Bundles.
- Architect data ingestion and transformation using DLT Expectations, modular Databricks Functions, and reusable pipeline components.
- Build scalable and modular ETL/ELT pipelines using PySpark.
- Lead implementation of Unity Catalog: create catalogs, schemas, role-based access policies, lineage visibility, and data classification tagging (PII, PHI, etc.).
- Guide organization-wide governance via Unity Catalog setup: workspace linkage, SSO, audit logging, external locations, and Volume access.
- Enable cross-platform data access using Lakehouse Federation, querying live from externally hosted databases.
- Leverage and integrate Databricks Marketplace to consume high-quality third-party data and publish internal data assets securely.
- Govern and manage Delta Sharing for securely sharing datasets with external partners or across tenants.
- Design and maintain PII anonymization, tokenization, and masking strategies using DBX functions and Unity Catalog policies to meet GDPR/HIPAA compliance.
- Architect Power BI, Tableau, and Looker integration with Databricks for live reporting and visualization over governed datasets.
- Build Databricks SQL Dashboards to enable stakeholders with real-time insights, KPI tracking, and alerts.
- Apply performance optimization techniques.
- Lead cross-functional initiatives across data science, analytics, and platform teams to deliver secure, scalable, and value-aligned data products.
- Provide thought leadership on adopting advanced features like Mosaic AI, Vector Search, Model Serving, and Databricks Marketplace publishing.
What you'll need
- Minimum 7.5 years of experience as a Data Architect and Data Engineer leading enterprise-scale Lakehouse initiatives.
- Must have skills: Databricks Unified Data Analytics Platform.
- Expert-level understanding of modern Data & Analytics Architecture patterns including Data Mesh, Data Products, and Lakehouse Architecture.
- Excellent programming and debugging skills in Python.
- Strong experience with PySpark for building scalable and modular ETL/ELT pipelines.
- Hands-on expertise in at least one major cloud platform: AWS, GCP, or Azure.
- Experience with cloud-based services relevant to data engineering, data storage, data processing, data warehousing, real-time streaming, and serverless computing.
- Hands-on experience applying performance optimization techniques.
- Strong background in data modeling and data warehousing concepts.
- 15 years full-time education.
Nice to have
- Working knowledge of DBT (Data Build Tool) is a plus.
- Databricks Certified Professional or similar certifications.
- Knowledge of machine learning concepts and experience with popular ML libraries.
- Knowledge of big data processing (e.g., Spark, Hadoop, Hive, Kafka).
- Data orchestration experience with Apache Airflow.
- Knowledge of CI/CD pipelines and DevOps practices in a cloud environment.
- Experience with ETL tools such as Informatica, Talend, Matillion, or Fivetran.
- Familiarity with DBT (Data Build Tool).
Details
Read the full description and apply on the company’s own careers page.