Overview
Senior Developer / Data Platform Engineer to design and build scalable data pipelines and modern Lakehouse solutions on Databricks-based platforms.
What you'll do
- Design and develop scalable data ingestion and transformation pipelines using PySpark and SQL.
- Build and manage batch and streaming ETL/ELT pipelines on Databricks.
- Design and maintain Lakehouse architecture using Bronze/Silver/Gold layers.
- Implement and optimize Apache Iceberg/open table formats, including schema evolution, partitioning, and compaction strategies.
- Design, monitor, and orchestrate end-to-end ingestion pipelines (batch + near real-time), including lineage and dependency management.
- Implement data quality checks, validation rules, reconciliation, monitoring, SLA tracking, and alerting.
What you'll need
- Strong experience in Databricks and PySpark ETL/ELT pipeline design and development.
- Experience with data ingestion and pipeline orchestration.
- SQL and distributed data processing skills.
- Experience with data quality and validation frameworks.
- Experience with Apache Iceberg / open table formats.
- Cloud experience (AWS preferred).
Nice to have
- Exposure to asset management / financial data domain.
- dbt (Data Build Tool).
- Data modeling (Data Vault / Star Schema exposure).
- Knowledge of Snowflake / Unity Catalog.
- CI/CD (Terraform, Git-based deployments).
Details
- Works on orchestration tools including Airflow and Databricks Jobs (or other schedulers).
- Implements CI/CD pipelines and automates deployment of Databricks jobs, notebooks, and configurations.
- Optimizes Spark jobs for cost, performance, and scalability.
- Follows data security, access control, and governance best practices including Unity Catalog / RBAC / metadata management.
Read the full description and apply on the company’s own careers page.
