Data Engineer

Overview

Design, build, maintain and analyze data pipelines and datasets that deliver actionable insights and support business decisions.

What you'll do

• Design, develop and maintain data solutions for data generation, collection and processing.
• Assist in the design and development of the data pipeline.
• Build data pipelines and ensure data quality by implementing ETL processes to migrate and deploy data across systems.
• Contribute to the design, development and implementation of data pipelines, ETL/ELT processes and data integration solutions.
• Design, develop and implement AI solutions using Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), vector search, tokenization, prompt engineering and AI-assisted development tools such as OpenAI Codex, GitHub Copilot, or equivalent platform.
• Take ownership of data pipeline projects from inception to deployment, managing scope, timelines and risks.
• Collaborate with cross-functional teams to understand data requirements and design solutions that meet business needs.
• Develop and maintain data models, data dictionaries and other documentation to ensure data accuracy and consistency.
• Implement data security and privacy measures to protect sensitive data.
• Leverage cloud platforms (AWS preferred) to build scalable and efficient data solutions.
• Collaborate and communicate effectively with product teams.
• Collaborate with Data Architects, Business SMEs and Data Scientists to design and develop end-to-end data pipelines to meet fast-paced business needs across geographic regions.
• Identify and resolve complex data-related challenges.
• Adhere to best practices for coding, testing and designing reusable code/components.
• Explore new tools and technologies that help improve ETL platform performance.
• Participate in sprint planning meetings and provide estimations on technical implementation.

What you'll need

• Master's or Bachelor's degree and 5-9 years of Computer Science, IT or related field experience.
• Hands-on experience with big data technologies and platforms, such as Databricks, Apache Spark (PySpark, SparkSQL), workflow orchestration and performance tuning on big data processing.
• Proficiency in data analysis tools (e.g. SQL) and experience with data visualization tools.
• Excellent problem-solving skills and the ability to work with large, complex datasets.
• Solid understanding of data governance frameworks, tools and best practices.
• Knowledge of data protection regulations and compliance requirements.
• Excellent critical-thinking and problem-solving skills.
• Good communication and collaboration skills.
• Demonstrated awareness of how to function in a team setting.
• Demonstrated presentation skills.

Nice to have

• Experience with ETL tools such as Apache Spark, and various Python packages related to data processing and machine learning model development.
• Good understanding of data modeling, data warehousing and data integration concepts.
• Knowledge of Python/R, Databricks, SageMaker and cloud data platforms.
• Hands-on experience designing, developing and implementing AI solutions using Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), vector search, tokenization, prompt engineering and AI-assisted development tools such as OpenAI Codex, GitHub Copilot, or equivalent platform.
• Professional certification as a Certified Data Engineer / Data Analyst (preferred on Databricks or cloud environments).

Details

• Location: Hyderabad, India.