Caterpillar Inc. logo

Senior Data Engineer

Caterpillar Inc.
NewPosted today

LOCATION

Bangalore · Onsite

EXPERIENCE

Not specified

TYPE

FullTime

SKILLS REQUIRED

Data PipelinesData QualityPythonAWS ServicesETL/ELTData ValidationAWS GluePySparkData Modeling

Job description

Overview

Design, build and maintain robust AWS data pipelines and connectors across AWS and Coveo, with a strong focus on data quality, reliability and validation, so that data ingested into Coveo and downstream systems is accurate, complete and production-ready.

What you'll do

  • Design, develop and maintain scalable data pipelines on AWS (batch and near-real-time).
  • Own data quality validation across pipelines, including completeness, freshness, consistency and accuracy checks.
  • Implement schema validation and anomaly detection.
  • Implement automated data quality checks, alerts and monitoring for pipeline failures or data issues.
  • Collaborate with search and platform teams to ensure high-quality indexed data for Coveo sources.
  • Contribute to CI/CD pipelines and follow enterprise SDLC and security standards.
  • Produce clear documentation for pipelines, data contracts and quality rules.
  • Perform in-depth analysis of search-indexed datasets to identify data quality issues and improve relevance, accuracy and overall search performance.

What you'll need

  • Strong experience as a Data Engineer working on AWS.
  • Hands-on experience building data pipelines using services such as AWS Glue, Bedrock, Lambda, Step Functions, S3 and SNS/SQS (or similar).
  • Proficiency in Python (and/or PySpark) for data engineering and validation logic.
  • Strong understanding of data quality concepts, frameworks and best practices.
  • Experience integrating data from multiple sources (APIs, databases, files).
  • Solid knowledge of ETL/ELT patterns and data modeling fundamentals.
  • Experience with logging, monitoring and alerting for data pipelines.
  • Familiarity with CI/CD practices and version control (Git).

Nice to have

  • Experience working with Coveo (sources, connectors, indexing, ingestion pipelines).
  • Prior experience supporting search platforms or content indexing pipelines.
  • Exposure to data quality frameworks or rule-based validation approaches.
  • Knowledge of Snowflake.
  • Experience working in large enterprise or multi-team environments.

Details

  • Location: Bangalore, Karnataka, India.

Read the full description and apply on the company’s own careers page.

Stay safe

Hiring on Abekus is free for applicants

We never charge a fee, and employers are prohibited from doing so. If a recruiter asks for payment, please report them right away.

Senior Data Engineer