Principal Data Engineer (PySpark, AI)

Overview

Principal Data Engineer on the Advanced Compliance Products (ACP) team, driving the architecture, scalability and modernization of ACP's data engineering platform used to detect non-compliant usage of Autodesk software.

What you'll do

• Define and drive the architecture of scalable, reliable and cost-efficient data platforms processing billions of records.
• Lead the design, development and optimization of large-scale data pipelines using Spark, PySpark, Python and SQL.
• Drive platform modernization initiatives, including incremental processing, reusable frameworks and modern data lakehouse architectures.
• Leverage AI/GenAI technologies and AI-assisted development tools to improve engineering productivity, automate workflows, enhance data quality and optimize pipeline operations.
• Identify and resolve complex performance, scalability, reliability and data quality challenges.
• Establish engineering best practices for CI/CD, automated testing, observability and cloud infrastructure.
• Lead technical design discussions, influence architectural decisions and mentor engineers across teams.
• Collaborate with product managers, data scientists and business stakeholders to translate business requirements into scalable data solutions.
• Take end-to-end technical ownership of complex, cross-team engineering initiatives.

What you'll need

• 10+ years of experience in data engineering, distributed systems or large-scale data processing.
• Strong hands-on expertise in Apache Spark, PySpark, Python, Spark SQL and SQL, including performance tuning and optimization.
• Proven experience architecting and scaling complex data pipelines and distributed data processing systems.
• Strong knowledge of AWS, Hive, Snowflake and modern data lakehouse architectures.
• Experience with data modeling, ETL/ELT frameworks, orchestration tools (such as Airflow) and CI/CD practices.
• Hands-on experience with AI/GenAI technologies, including LLMs, prompt engineering and AI-assisted development tools such as GitHub Copilot, Cursor, or equivalent.
• Practical experience applying AI to data engineering workflows, such as code generation, automated testing, data quality validation, anomaly detection, pipeline optimization or intelligent troubleshooting.
• Strong understanding of data quality, observability, reliability and cloud infrastructure.
• Demonstrated technical leadership, architectural decision-making and mentoring experience.
• Excellent problem-solving, communication and cross-functional collaboration skills.
• Bachelor's degree in Computer Science, Engineering or a related field, or equivalent practical experience.

Nice to have

• Experience with Apache Iceberg, incremental data processing and modern orchestration frameworks.
• Experience modernizing legacy data platforms and optimizing large-scale Spark workloads.
• Familiarity with BI tools such as Power BI or Looker.
• Experience with AI agents or LLM-powered automation.
• Experience building reusable data engineering frameworks and driving engineering standards across teams.

Details

• Location: Bengaluru, India.
• May require in-person onboarding and/or in-person ID verification.