Architect

Overview

Join Twilio's Data Substrate team as an Architect to own the technical strategy for the company's data platform, spanning storage, lake and lakehouse architecture, catalog and governance, and privacy-by-design.

What you'll do

• Own the architecture and multi-year technical strategy for the data platform, including the lake/lakehouse, ClickHouse, Elasticsearch, and governance, keeping them coherent as one platform rather than separate systems.
• Define where each engine fits (for example, ClickHouse versus open table formats on object storage) and the migration path between them.
• Make and document major design decisions (ADRs, RFCs) on storage and open table formats, catalog strategy, compute engines, tenancy, data placement, and build-versus-adopt tradeoffs, with cost and performance at petabyte scale as explicit design constraints.
• Set reference architectures for how core storage and query engines (ClickHouse, Elasticsearch, and lake engines) are deployed, tuned, scaled and operated as shared platforms, including capacity, cost, tenant isolation, and upgrade strategy.
• Define the reliability model for the platform: availability targets, failure domains, disaster recovery, observability, and incident response patterns that teams can adopt.
• Build privacy and compliance into the architecture, not onto it, designing deletion and right-to-erasure, retention, data residency, and PII classification to work across storage layers, backups, derived datasets, and data arriving from upstream streaming systems.
• Drive data governance: classification, catalog and metadata, lineage, access control integrated with authentication and authorization systems, data contracts, quality and integrity standards, and a catalog teams actually use.
• Review designs and technical direction across the org, and act as the escalation point on hard distributed-systems and performance problems.
• Mentor principal and senior engineers, and grow architectural judgment across the org, not only within one squad.
• Partner with product, security, infrastructure and consuming teams to turn business needs into platform roadmaps, and say no or re-sequence when tradeoffs require it.
• Track emerging data technologies (such as streaming-first architectures, federated query, and AI-ready data layers) and bring forward those worth adopting, with evidence from prototypes or benchmarks.

What you'll need

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field, or equivalent practical experience.
• 15+ years of experience in software development or a related field, with at least 8 years in an architecture or senior technical leadership role.
• Proven track record of architecting and delivering large-scale data platforms and setting their technical strategy, on cloud infrastructure such as AWS, deployed with Kubernetes, Terraform and Helm.
• Deep expertise in storage and compute for analytical data: columnar and open table formats, lake and lakehouse architecture, partitioning, indexing, query planning, and how the choices affect cost and performance at petabyte scale.
• Breadth across data storage and retrieval technologies, with hands-on depth in at least two of: analytical/OLAP engines (e.g., ClickHouse), search engines (e.g., ElasticSearch/OpenSearch), and lake or distributed processing engines (e.g., Spark, Trino, Flink); able to reason about when to use each, how they interoperate, and how to migrate between them.
• Working knowledge of streaming systems (e.g., Kafka) and how they interact with downstream storage.
• Practical experience designing data governance at scale: data classification, catalog and metadata management, lineage, and access control integrated with authentication and authorization systems.
• Practical experience implementing privacy and compliance requirements (deletion, retention, residency) across a large, heterogeneous data estate.
• Proficiency in one or more of Python, Java, and Go.
• Track record of writing and driving architecture decisions across multiple teams, and of changing the direction of an organization's technical roadmap through influence rather than authority.
• Experience elevating engineering standards and mentoring senior engineers across teams.
• Demonstrated commitment to operational excellence: architecting for high availability, observability and incident response in distributed systems, and being credible when a platform is failing in production.
• Strong written and verbal communication, including making a complex tradeoff clear to engineers and to senior leadership.

Nice to have

• Experience operating one of these engines (e.g., ClickHouse, ElasticSearch) at scale, including replication, sharding, ingestion behavior, and multi-tenant isolation.
• Experience with catalog and metadata systems (e.g., Unity Catalog, Glue, Polaris, DataHub) and open table format catalog strategy.
• Experience with a platform migration across domains or ownership models, such as moving from central to domain-owned data.
• A track record of measurable cost optimization on large data platforms.
• Networking and network security knowledge.

Details

• Remote role based in the United States.
• May be required to travel occasionally for project or team in-person meetings.
• May be asked to report in person on an ad-hoc basis for team gatherings, functional off-sites or customer meetings.
• Full-time role eligible for Twilio's equity plan and corporate bonus plan; generally eligible for health care, 401(k), paid sick time, paid personal time off, and paid parental leave.
• Applications intended to be accepted until 1/4/2027, subject to change based on business needs.