Senior Data Engineer

Overview

Senior Data Engineer to design and operate a data platform, build production data pipelines, and support AI-retrieval and generative AI capabilities.

What you'll do

• Design scalable distributed data processing systems for the data platform.
• Build ETL/ELT pipelines for complex datasets.
• Engineering high-quality datasets for AI models.
• Implement retrieval architectures and generative AI capabilities.
• Develop backend services to expose data products.
• Optimize code performance, data quality, and storage efficiency.
• Guide junior engineers and advocate technical best practices.

What you'll need

• 7+ years of professional software engineering experience.
• Expert-level data engineering with Apache Spark, Hadoop, Kafka, and Databricks/Delta Lake.
• Python ecosystem experience including PySpark, Pandas, and NumPy.
• Advanced cloud deployment experience in AWS or Azure.
• Expert knowledge of relational and non-relational databases for data modeling.
• Experience building and shipping production-grade data pipelines.
• Bachelor’s degree in Computer Science or a related field.

Nice to have

• Experience with LangChain, LlamaIndex, and vector databases such as Pinecone and Milvus.
• Experience building APIs using FastAPI, Flask, or Node.js.
• Containerization and CI/CD workflows using Docker, Kubernetes, and related tools.