Irth Solutions logo

Data Scientist ML/Gen AI Engineer

Irth Solutions
NewPosted today

LOCATION

India · Remote

EXPERIENCE

3 - 6 Years

TYPE

FullTime

SKILLS REQUIRED

Machine LearningPythonDatabricksRetrieval-Augmented GenerationSQLSparkLLM Evaluation

Job description

Overview

Hands-on ML/GenAI Engineer in the Insights (AI/ML) team, building and productionizing machine learning and GenAI solutions on Irth's governed Databricks Lakehouse, reporting to the Data Platform & Analytics Manager.

What you'll do

  • Contribute to medallion architecture pipelines (Bronze → Silver → Gold) using Databricks.
  • Implement data quality checks, validation gates, and data contracts at ingestion.
  • Support column-level lineage and governance initiatives, targeting at least 95% lineage coverage.
  • Help implement policy-as-code for regional data residency and sensitive-data handling.
  • Ensure appropriate PII masking, obfuscation, and access controls across Silver and Gold data layers.
  • Collaborate with data engineering and governance teams to improve data reliability, discoverability, and documentation.
  • Explore, prototype, evaluate, and productionize machine learning and GenAI solutions, including forecasting, anomaly detection, NLP, RAG, LLM-powered assistants and copilots, and predictive analytics.
  • Develop solutions that address measurable customer and business problems across Irth's industry verticals.
  • Package and manage models using Unity Catalog model management/registries.
  • Design and implement batch and streaming inference architectures where appropriate.
  • Partner with Product and business stakeholders to define success metrics, KPIs, and A/B testing strategies.
  • Move successful experiments from prototype to production with clearly defined SLAs, monitoring, documentation, and operational runbooks.
  • Build production workflows, jobs, and notebooks as assets-as-code using Databricks Asset Bundles (DABs).
  • Implement CI/CD pipelines using GitHub Actions.
  • Design reliable, observable, and scalable data and ML workloads, working toward SLOs including pipeline success rate ≥99.5%, P1 Mean Time to Detect ≤5 minutes, and Mean Time to Repair ≤60 minutes.
  • Implement proactive monitoring and alerting.
  • Automate incident creation and tracking through Jira where appropriate.
  • Apply FinOps principles, including resource tagging, workload policies, optimization, and cost monitoring.
  • Identify opportunities to improve compute performance while maintaining cost efficiency.
  • Contribute business metrics, definitions, and semantic models to Unity Catalog and help establish a single source of truth for metrics.
  • Support consumption through Power BI and Databricks AI/BI.
  • Work with domain teams to develop and maintain trusted data products, and ensure analytical definitions remain consistent across products and business functions.
  • Implement secure data and ML architectures using RBAC and ABAC within Unity Catalog, following secure networking practices.
  • Manage credentials and secrets using cloud key-management and secret-management services, such as Azure Key Vault (AKV) or KMS.
  • Design solutions with security, privacy, and auditability built into the development lifecycle.
  • Support compliance requirements across SOC 2, ISO 27001, GDPR, and PIPEDA, and produce and maintain audit evidence (data lineage, access reviews, data retention, security controls, disaster recovery testing and drills).
  • Participate in governance and security reviews and remediate identified gaps.

What you'll need

  • 3–6 years of experience in Data Science, Machine Learning, or ML Engineering, with a proven track record of taking models from development through production.
  • Strong programming and data skills in Python, SQL, and Spark/PySpark.
  • Hands-on experience with Databricks, including Delta Lake, Unity Catalog, Databricks SQL (DBSQL), Jobs and Workflows, and medallion architecture.
  • Strong understanding of ML fundamentals, including feature engineering, model training and selection, model evaluation and validation, model monitoring, data-quality monitoring, and model and data drift detection.
  • Practical GenAI/LLM experience, including prompt engineering, Retrieval-Augmented Generation (RAG), vector databases/vector stores, LLM evaluation, and AI safety and guardrails.
  • Understanding of LLM latency, scalability, and cost tradeoffs.
  • Experience implementing CI/CD for data and ML workloads, including GitHub Actions, Databricks Asset Bundles (DABs), DEV → QA → PROD environment promotion, and secrets and configuration management.
  • Experience with data contracts and data-quality frameworks, including schema governance, automated expectations/testing, validation, and quarantine/error-handling workflows.
  • Strong understanding of data security and compliance, including PII handling and protection, RBAC/ABAC, data residency requirements, and policy-as-code.
  • Strong communication and collaboration skills, with the ability to work effectively with Product, Engineering, Data, and domain teams.
  • Ability to produce clear technical documentation, including Architecture Decision Records (ADRs), runbooks, experiment reports, and operational documentation.

Nice to have

  • Experience with Microsoft Azure, including Azure Data Lake Storage (ADLS), Azure Active Directory / Microsoft Entra ID, Azure Key Vault (AKV), Microsoft Fabric, and Power BI.
  • Experience with AWS, including Amazon S3, AWS KMS, AWS Secrets Manager, Amazon RDS, and DynamoDB.
  • Experience with geospatial data and analytics, including PostGIS, spatial joins, spatial indexing, tiling, and GIS-based feature engineering.
  • Experience with streaming and real-time data, including Structured Streaming and Change Data Capture (CDC).
  • Hands-on experience with MLflow and Unity Catalog Model Serving.
  • Experience implementing data and ML observability, including model performance metrics, lineage dashboards, pipeline monitoring, SLA/SLO monitoring, and alerting.
  • Understanding of FinOps practices, including resource tagging, budgets, cost monitoring, and cost anomaly detection.
  • Familiarity with Disaster Recovery (DR), Business Continuity Planning (BCP), and resilience practices.
  • Experience working in utilities, energy, infrastructure, public works, or related industries.
  • Experience building predictive, risk-scoring, or failure-prediction models for asset integrity, including corrosion, defects, degradation, or infrastructure failure.
  • Experience applying anomaly detection and time-series forecasting to pipeline inspection, sensor, maintenance, or operational data.
  • Experience engineering ML features from GIS and geospatial asset data, including pipeline routes, facilities, inspection locations, and infrastructure networks.
  • Experience developing risk models using pipeline, utility, or asset-integrity data.
  • Understanding of regulatory, compliance, and audit-reporting requirements associated with asset integrity and infrastructure analytics.
  • Experience translating analytical and ML outputs into operational risk indicators, customer-facing insights, or decision-support tools.

Details

  • Location: Remote – India.
  • Department: Insights (AI/ML); reports to Data Platform & Analytics Manager.

Read the full description and apply on the company’s own careers page.

Stay safe

Hiring on Abekus is free for applicants

We never charge a fee, and employers are prohibited from doing so. If a recruiter asks for payment, please report them right away.

Data Scientist ML/Gen AI Engineer