Senior Data , BI & AI Enablement Engineer

Overview

Design, build and maintain the data infrastructure, pipelines and data products that enable reliable data integration, transformation and delivery across the enterprise, including Databricks administration.

What you'll do

• Design, develop and maintain robust data pipelines using Databricks and Apache Spark to extract, transform and load data into appropriate storage systems.
• Implement and manage advanced data products in the medallion architecture.
• Implement and manage advanced data models, including dimensional, relational and Data Vault modeling techniques.
• Integrate data from databases, data warehouses, APIs and external systems.
• Ensure data consistency and integrity during integration, performing data validation and cleaning as needed.
• Transform raw data into usable format by applying cleansing, aggregation, filtering and enrichment techniques.
• Optimize data pipelines and processing workflows for performance, scalability and efficiency.
• Monitor and tune data systems, identify and resolve performance bottlenecks, and implement caching and indexing strategies.
• Implement data quality checks and validations within pipelines to ensure accuracy, consistency and completeness.
• Manage Databricks Unity Catalog, including administrative activities (policies, security, access control, cost control).
• Optimize and administer Databricks environments for high performance and reliability, including cluster configuration and policy management.
• Build and maintain CI/CD pipelines (DevOps) in GitHub or Azure DevOps.
• Collaborate with cross-functional teams to deliver comprehensive data solutions.
• Ensure data quality, security and governance across all data processes.
• Communicate effectively with stakeholders to understand requirements and deliver actionable insights.
• Operate within agile teams, contributing to continuous improvement in data engineering practices and processes.
• Utilize cloud platforms such as AWS and Azure to build scalable data solutions.
• Collaborate with data scientists and data analysts to optimize models and algorithms for data quality, security and governance.

What you'll need

• Bachelor's degree in computer science, data science, software engineering, information systems, information technology or a related quantitative field.
• Minimum of 6+ years of experience in data engineering, with at least 2+ years of hands-on experience with Databricks.
• At least 6+ years of work experience in data management disciplines, including data integration, modeling, optimization and data quality, or other areas directly relevant to data engineering.
• Strong knowledge of SQL, Python and PySpark.
• Proficiency in modern data modeling techniques and data administration.
• Experience with cloud platforms such as AWS and Azure.
• Expert problem-solving skills, including debugging in unfamiliar code or systems, and the ability to recognize and solve repetitive problems.
• Strong communication skills and a proactive, "getting things done" mindset.
• Experience working in agile teams and familiarity with agile methodologies.
• Ability to design, build and deploy data solutions that capture, explore, transform and utilize data to support AI, ML and BI.
• Proficiency in the design and implementation of modern data architectures and concepts such as cloud services (AWS, Azure, GCP) and modern data warehouse tools (Snowflake, Databricks).
• Experience with database technologies such as SQL, NoSQL, Oracle, Hadoop or Teradata.
• Ability to collaborate across teams of different technical knowledge and educate end users on data products.
• Excellent business acumen and interpersonal skills; able to work across business lines at a senior level to influence and effect change.
• Ability to describe business use cases/outcomes, data sources, management concepts and analytical approaches/options.
• Ability to translate among the languages used by executive, business, IT and quant stakeholders.

Nice to have

• Databricks certifications (Associate, Professional, Administration).
• Master's degree.