Principal Data Engineer – Python Backend

Overview

Data Engineer role combining data engineering with Python backend API development, including integrating data platforms with applications and AI agents.

What you'll do

• Design, build, and maintain data ingestion and ETL/ELT pipelines using Python, SQL, Databricks, and Spark/PySpark.
• Integrate data from APIs, databases, files, and third-party systems.
• Develop and maintain backend services and REST APIs using Python frameworks such as FastAPI, Flask, or Django.
• Build API integrations that allow applications and AI agents to access approved data and services.
• Design and work with relational and NoSQL databases, including schemas, data models, and queries.
• Optimize Spark jobs and Databricks workloads for performance, scalability, and cost.
• Apply distributed computing principles to process large datasets reliably.
• Implement data validation, error handling, logging, and monitoring.
• Troubleshoot pipeline, API, and database issues in development and production environments.
• Write tests and documentation, and participate in Git, code reviews, CI/CD, and deployment workflows.
• Collaborate with data, backend, AI/ML, product, and application teams.

What you'll need

• Master's OR Bachelor's degree in IT/Computer Science/Engineering.
• 12 to 17 years of work experience.
• Strong Python development skills.
• Strong SQL skills and experience with relational databases.
• Hands-on experience with Databricks and Apache Spark/PySpark.
• Experience building data pipelines and integrating data from multiple sources.
• Experience developing or integrating REST APIs.
• Working knowledge of NoSQL databases.
• Hands-on experience integrating APIs with AI agents or AI/ML applications.
• Understanding of Spark performance optimization, scalability, and distributed systems.
• Familiarity with Git, testing, debugging, and production support.
• Strong problem-solving and communication skills.

Nice to have

• Experience with FastAPI, Flask, or Django.
• Experience with Delta Lake, Azure Data Lake, or Azure Data Factory.
• Familiarity with Airflow, Kafka, Docker, or Kubernetes.
• Experience with authentication and authorization for APIs.
• Experience supporting GenAI or machine learning data pipelines.