Senior Data Engineer

Overview

Design, build and maintain scalable data platforms and data processing pipelines using Python, PySpark, AWS, DynamoDB, OpenSearch, and Java exposure, with a focus on cloud-native big data solutions.

What you'll do

• Design, develop and maintain large-scale data pipelines using Python and PySpark.
• Build and optimize batch and near real-time data processing solutions.
• Design scalable data architectures on AWS.
• Develop ETL/ELT workflows for ingesting, transforming and publishing datasets.
• Work with structured and semi-structured data from multiple sources.
• Design and maintain NoSQL database solutions using DynamoDB.
• Implement search and analytics solutions using OpenSearch.
• Optimize data processing jobs for performance, scalability and cost efficiency.
• Collaborate with product teams, architects and software engineers to define data requirements.
• Monitor, troubleshoot and resolve data platform and pipeline issues.
• Implement data quality, governance, security and observability best practices.
• Participate in code reviews and mentor junior engineers.

What you'll need

• Strong expertise in Python.
• Extensive experience with PySpark and distributed data processing.
• Hands-on experience building data lakes and modern data platforms.
• Strong understanding of ETL/ELT frameworks and data modeling.
• Experience processing large datasets efficiently.
• Expert-level knowledge of AWS services such as EMR, Glue, Lambda, S3, Athena, Redshift, ECS/EKS, Step Functions, EventBridge, IAM, CloudWatch and SNS/SQS.
• Strong SQL skills and query optimization experience.
• Experience with NoSQL databases, particularly DynamoDB.
• Understanding of partitioning, indexing and data access patterns.
• Hands-on experience with OpenSearch / Elasticsearch.
• Experience designing indexes and search schemas.
• Knowledge of query optimization and search performance tuning.
• Experience implementing analytics and monitoring dashboards.
• Strong Python development skills.
• Working knowledge of Java.
• Understanding of object-oriented programming and software engineering best practices.
• BA/BS degree and 8-10 years' relevant experience, OR equivalent combination of education and experience.

Nice to have

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• 8+ years of experience in Data Engineering.
• Experience with streaming platforms such as Kafka or Kinesis.
• Experience with Airflow or other workflow orchestration tools.
• Experience with Docker and Kubernetes.
• Exposure to Data Lake and Lakehouse architecture.
• Experience in Agile/Scrum development environments.
• Apache Kafka.
• Apache Airflow.
• Terraform.
• Databricks.
• Delta Lake.
• Snowflake.
• CI/CD pipelines.
• Machine Learning data pipelines.