Overview
Embedded architectural advisor for a portfolio of stream-aligned data domain teams within FactSet's Data Engineering organization, supporting the Data Platform initiative and closing the architectural coverage gap created by a growing number of stream-aligned teams ingesting data into the platform.
What you'll do
- Serve as the embedded architectural advisor for a portfolio of stream-aligned data domain teams, developing deep expertise in the unique data characteristics, source systems, SLAs, volumes, and business rules of each domain.
- Conduct discovery and design sessions with domain engineering teams to translate business-level data requirements into concrete pipeline architectures before implementation begins.
- Design domain-specific ingestion pipeline architectures on Databricks, including medallion layer structure, Delta Lake table design, incremental processing patterns, and Unity Catalog organization, that conform to platform standards while accommodating legitimate domain variation.
- Champion domain use cases back to the platform teams, translating concrete domain requirements into platform backlog items, paved-road improvements, and ADRs.
- Guide domain teams through adoption of platform-standard tooling.
- Identify repeated patterns across domain teams and work with platform teams to elevate them into reusable templates, accelerators, and paved roads that reduce onboarding cost for subsequent teams.
- Develop and maintain domain-level architecture decision records, pipeline reference architectures, and onboarding runbooks that domain engineers can use independently.
- Participate in architecture reviews across domain projects and provide structured feedback against platform principles, quality gates, and certification criteria.
- Collaborate with the platform to ensure pipeline designs incorporate data quality checks and observability instrumentation aligned with the platform's certification framework.
- Contribute to the evolution of platform ingestion standards and guardrails, ensuring they remain practical and actionable for the stream-aligned teams who must implement them.
- Mentor senior engineers within domain teams and contribute to the broader technical community across the Data Platform.
What you'll need
- 10+ years of software engineering experience with deep expertise in data engineering or data architecture, with a proven track record of designing and delivering production data platforms at scale.
- Experience in financial services space.
- Hands-on experience in creating data models and taxonomies.
- Deep hands-on expertise with Databricks, including Databricks Workflows, Delta Lake, and Unity Catalog, sufficient to design, review, and guide production pipeline implementations.
- Strong proficiency in Python and SQL for pipeline design, proof-of-concept implementation, and design review.
- Extensive experience designing layered / medallion pipeline architectures and incremental processing strategies including CDC, streaming ingestion, micro-batch, and merge/upsert patterns.
- Demonstrated ability to engage technically with stream-aligned or cross-functional engineering teams and translate business data requirements into concrete architectural designs.
- Experience producing architecture artifacts that other engineering teams can adopt and maintain independently such as ADRs, reference architectures, pipeline blueprints, and design patterns.
- Strong collaboration skills, with the ability to align diverse stakeholders across domain teams and platform teams.
- Track record of mentoring senior engineers and growing technical capabilities within delivery teams.
- Ability to operate at multiple altitudes: hands-on pipeline design with domain engineers and clear articulation of platform impact and trade-offs to senior architects and leadership.
Nice to have
- Experience with Open Tables and declarative pipeline frameworks on Databricks or Snowflake.
- Familiarity with streaming platforms (Apache Kafka, AWS Kinesis, Confluent) and patterns for integrating event streams with Databricks lakehouse architectures.
- Experience with Unity Catalog for data governance, lineage, and fine-grained access control across a multi-domain lakehouse.
- Experience with open table formats (Delta Lake, Apache Iceberg, Apache Hudi) and their operational characteristics, including compaction, time travel, and schema evolution, at production scale.
- Familiarity with AWS-based data infrastructure (S3, Glue, EMR, Kinesis) as a pipeline substrate or source-system integration layer.
- Experience onboarding diverse source systems (relational databases, event streams, flat files, external APIs) into a lakehouse architecture.
- Background in data domain modeling and appreciation for what makes different data product areas distinct (e.g., pricing data, reference/master data, risk analytics, market data).
- Familiarity with data contracts, versioned schemas, and the role these play in decoupling pipeline producers from downstream consumers.
- Understanding of data mesh or data fabric architectural patterns and their practical implications for a federated, domain-oriented platform.
- Experience with Apache Spark and distributed data processing at scale.
- Background in cost optimization for cloud-based data processing, including storage tiering, partition design, and compute scaling strategies.
- Experience working in financial services or with financial market data systems.
- Familiarity with DataOps practices and platform engineering approaches for data infrastructure.
- Certifications in Databricks, AWS, or related technologies.
Details
- Location: Hyderabad, India.
Read the full description and apply on the company’s own careers page.