Arcfront logo

Platform Engineer

Arcfront
NewPosted today

LOCATION

Bengaluru · Onsite

EXPERIENCE

Not specified

TYPE

FullTime

SKILLS REQUIRED

Data PipelinesDistributed SystemsSystem DesignPerformance OptimizationFault ToleranceData Privacy

Job description

Overview

Platform Engineer role owning the large-scale data systems at the core of Arcfront, moving, cleaning and protecting very large datasets so they can become high-quality training data.

What you'll do

  • Own big-data engineering and architecture for pipelines that handle very large datasets (often 10TB+).
  • Design pipelines with fault tolerance built in from day one: checkpointing, idempotency, dead-letter queues and graceful handling of partial failures.
  • Partition and shard large datasets sensibly across pipelines, and reason about batching and fan-out.
  • Build and improve PII redaction, including the evaluations that measure redaction quality and produce accuracy scores.
  • Design pipelines that can onboard many new data sources in short timeframes, and estimate the infrastructure cost of doing so.
  • Own the long-term architecture and sustainability of the data platform.

What you'll need

  • Strong foundation in data structures and algorithms, including reasoning about behaviour at very large scale (billions of rows).
  • Hands-on experience with at least one of: Spark, Kafka, Airflow, Flink or Ray.
  • Working knowledge of distributed processing: partitioning, DAGs and shuffle cost, executor memory sizing, and handling out-of-memory failures.
  • Familiarity with storage trade-offs: columnar vs row formats (e.g. Parquet), partitioning strategy, hot vs cold data.
  • Comfortable using LLMs, and with building evals to measure the quality of data processing such as PII redaction.

Read the full description and apply on the company’s own careers page.

Stay safe

Hiring on Abekus is free for applicants

We never charge a fee, and employers are prohibited from doing so. If a recruiter asks for payment, please report them right away.

Platform Engineer