Overview
Hands-on ML/GenAI Engineer in the Insights (AI/ML) team, building and productionizing machine learning and GenAI solutions on Irth's governed Databricks Lakehouse, reporting to the Data Platform & Analytics Manager.
What you'll do
- Contribute to medallion architecture pipelines (Bronze → Silver → Gold) using Databricks.
- Implement data quality checks, validation gates, and data contracts at ingestion.
- Support column-level lineage and governance initiatives, targeting at least 95% lineage coverage.
- Help implement policy-as-code for regional data residency and sensitive-data handling.
- Ensure appropriate PII masking, obfuscation, and access controls across Silver and Gold data layers.
- Collaborate with data engineering and governance teams to improve data reliability, discoverability, and documentation.
- Explore, prototype, evaluate, and productionize machine learning and GenAI solutions, including forecasting, anomaly detection, NLP, RAG, LLM-powered assistants and copilots, and predictive analytics.
- Develop solutions that address measurable customer and business problems across Irth's industry verticals.
- Package and manage models using Unity Catalog model management/registries.
- Design and implement batch and streaming inference architectures where appropriate.
- Partner with Product and business stakeholders to define success metrics, KPIs, and A/B testing strategies.
- Move successful experiments from prototype to production with clearly defined SLAs, monitoring, documentation, and operational runbooks.
- Build production workflows, jobs, and notebooks as assets-as-code using Databricks Asset Bundles (DABs).
- Implement CI/CD pipelines using GitHub Actions.
- Design reliable, observable, and scalable data and ML workloads, working toward SLOs including pipeline success rate ≥99.5%, P1 Mean Time to Detect ≤5 minutes, and Mean Time to Repair ≤60 minutes.
- Implement proactive monitoring and alerting.
- Automate incident creation and tracking through Jira where appropriate.
- Apply FinOps principles, including resource tagging, workload policies, optimization, and cost monitoring.
- Identify opportunities to improve compute performance while maintaining cost efficiency.
- Contribute business metrics, definitions, and semantic models to Unity Catalog and help establish a single source of truth for metrics.
- Support consumption through Power BI and Databricks AI/BI.
- Work with domain teams to develop and maintain trusted data products, and ensure analytical definitions remain consistent across products and business functions.
- Implement secure data and ML architectures using RBAC and ABAC within Unity Catalog, following secure networking practices.
- Manage credentials and secrets using cloud key-management and secret-management services, such as Azure Key Vault (AKV) or KMS.
- Design solutions with security, privacy, and auditability built into the development lifecycle.
- Support compliance requirements across SOC 2, ISO 27001, GDPR, and PIPEDA, and produce and maintain audit evidence (data lineage, access reviews, data retention, security controls, disaster recovery testing and drills).
- Participate in governance and security reviews and remediate identified gaps.
What you'll need
- 3–6 years of experience in Data Science, Machine Learning, or ML Engineering, with a proven track record of taking models from development through production.
- Strong programming and data skills in Python, SQL, and Spark/PySpark.
- Hands-on experience with Databricks, including Delta Lake, Unity Catalog, Databricks SQL (DBSQL), Jobs and Workflows, and medallion architecture.
- Strong understanding of ML fundamentals, including feature engineering, model training and selection, model evaluation and validation, model monitoring, data-quality monitoring, and model and data drift detection.
- Practical GenAI/LLM experience, including prompt engineering, Retrieval-Augmented Generation (RAG), vector databases/vector stores, LLM evaluation, and AI safety and guardrails.
- Understanding of LLM latency, scalability, and cost tradeoffs.
- Experience implementing CI/CD for data and ML workloads, including GitHub Actions, Databricks Asset Bundles (DABs), DEV → QA → PROD environment promotion, and secrets and configuration management.
- Experience with data contracts and data-quality frameworks, including schema governance, automated expectations/testing, validation, and quarantine/error-handling workflows.
- Strong understanding of data security and compliance, including PII handling and protection, RBAC/ABAC, data residency requirements, and policy-as-code.
- Strong communication and collaboration skills, with the ability to work effectively with Product, Engineering, Data, and domain teams.
- Ability to produce clear technical documentation, including Architecture Decision Records (ADRs), runbooks, experiment reports, and operational documentation.
Nice to have
- Experience with Microsoft Azure, including Azure Data Lake Storage (ADLS), Azure Active Directory / Microsoft Entra ID, Azure Key Vault (AKV), Microsoft Fabric, and Power BI.
- Experience with AWS, including Amazon S3, AWS KMS, AWS Secrets Manager, Amazon RDS, and DynamoDB.
- Experience with geospatial data and analytics, including PostGIS, spatial joins, spatial indexing, tiling, and GIS-based feature engineering.
- Experience with streaming and real-time data, including Structured Streaming and Change Data Capture (CDC).
- Hands-on experience with MLflow and Unity Catalog Model Serving.
- Experience implementing data and ML observability, including model performance metrics, lineage dashboards, pipeline monitoring, SLA/SLO monitoring, and alerting.
- Understanding of FinOps practices, including resource tagging, budgets, cost monitoring, and cost anomaly detection.
- Familiarity with Disaster Recovery (DR), Business Continuity Planning (BCP), and resilience practices.
- Experience working in utilities, energy, infrastructure, public works, or related industries.
- Experience building predictive, risk-scoring, or failure-prediction models for asset integrity, including corrosion, defects, degradation, or infrastructure failure.
- Experience applying anomaly detection and time-series forecasting to pipeline inspection, sensor, maintenance, or operational data.
- Experience engineering ML features from GIS and geospatial asset data, including pipeline routes, facilities, inspection locations, and infrastructure networks.
- Experience developing risk models using pipeline, utility, or asset-integrity data.
- Understanding of regulatory, compliance, and audit-reporting requirements associated with asset integrity and infrastructure analytics.
- Experience translating analytical and ML outputs into operational risk indicators, customer-facing insights, or decision-support tools.
Details
- Location: Remote – India.
- Department: Insights (AI/ML); reports to Data Platform & Analytics Manager.
Read the full description and apply on the company’s own careers page.