Software Development Engineer Staff

Overview

Staff-level GCP Data Engineer on the Equinix Data Team, responsible for end-to-end development of data engineering use cases and the Equinix Data Lake platform and tools.

What you'll do

• Design, develop, and maintain scalable ETL/ELT pipelines using Cloud Dataflow, Cloud Composer (Apache Airflow), Dataform/dbt and Cloud Functions.
• Build real-time streaming data pipelines using Cloud Pub/Sub, Kafka and Dataflow.
• Implement automated data quality checks and monitoring across all data workflows.
• Optimize pipeline performance and cost efficiency through proper resource allocation and scheduling.
• Architect and implement data lake and data warehouse solutions using Dataproc, BigQuery, Cloud Storage, and Cloud SQL.
• Design optimal data models, partitioning strategies, and clustering for analytical workloads.
• Manage data lifecycle policies and implement automated archival and retention strategies.
• Ensure data security, encryption, and access control across all storage layers.
• Build and optimize BigQuery datasets for analytics and reporting use cases.
• Create and maintain dimensional models and fact tables for business intelligence.
• Implement data marts and aggregation layers for improved query performance.
• Support self-service analytics through proper data cataloging and documentation.
• Integrate data from databases, APIs, SaaS applications, and file systems, including third-party data providers and external data feeds.
• Implement change data capture (CDC) solutions for real-time data synchronization.
• Implement comprehensive monitoring and alerting using Cloud Monitoring and Cloud Logging.
• Troubleshoot data pipeline issues and implement robust error handling mechanisms.
• Maintain data lineage documentation and impact analysis capabilities.

What you'll need

• 4+ years hands-on experience with BigQuery, Cloud Dataflow, Cloud Composer, Cloud Storage, Cloud Pub/Sub, Dataform/dbt and Cloud Functions.
• Proficiency in Python/Java and SQL; experience with Spark/Beam development.
• Strong understanding of Apache Beam, Apache Airflow, and distributed computing concepts.
• Experience with both RDBMS (Cloud SQL, PostgreSQL, MySQL) and NoSQL (Bigtable, Firestore) databases.
• Experience with Terraform, Cloud Deployment Manager, or similar infrastructure as code tools.
• Bachelor's degree in computer science, Engineering, or related field.
• 4+ years of experience in data engineering or related roles.
• 4+ years of specific experience with Google Cloud Platform.
• Experience with version control systems (Git) and CI/CD pipelines.
• Knowledge of data modelling techniques and dimensional modelling.

Nice to have

• Google Cloud Professional Data Engineer certification.
• Experience with containerization (Docker, Kubernetes/GKE).
• Experience with data governance and compliance frameworks (GDPR, HIPAA, SOX).
• Familiarity with business intelligence tools (Looker, Tableau, Power BI).
• Experience with streaming technologies beyond GCP (Kafka, Spark Streaming).

Details

• Location: Bangalore Office BLS2.
• Good knowledge of Dataplex and Analytics Hub is listed as a responsibility.