Senior Associate GCP Data Engineer

Overview

Serve as a Senior Associate GCP Data Engineer on the Data Team, developing end-to-end data engineering use cases and supporting the Equinix Data Lake platform and tools. Design, build, and maintain scalable data infrastructure and analytics solutions on Google Cloud Platform.

What you'll do

• Design, develop, and maintain scalable ETL/ELT pipelines using Cloud Dataflow, Cloud Composer (Apache Airflow), Dataform/dbt, and Cloud Functions.
• Build real-time streaming data pipelines using Cloud Pub/Sub, Kafka, and Dataflow.
• Implement automated data quality checks and monitoring across all data workflows.
• Optimize pipeline performance and cost efficiency through proper resource allocation and scheduling.
• Architect and implement data lake and data warehouse solutions using Dataproc, BigQuery, Cloud Storage, and Cloud SQL.
• Design optimal data models, partitioning strategies, and clustering for analytical workloads.
• Manage data lifecycle policies and implement automated archival and retention strategies.
• Ensure data security, encryption, and access control across all storage layers.
• Build and optimize BigQuery datasets for analytics and reporting use cases.
• Create and maintain dimensional models and fact tables for business intelligence.
• Implement data marts and aggregation layers for improved query performance.
• Support self-service analytics through proper data cataloging and documentation.
• Integrate data from databases, APIs, SaaS applications, and file systems.
• Implement change data capture (CDC) solutions for real-time data synchronization.
• Work with third-party data providers and external data feeds.
• Implement comprehensive monitoring and alerting using Cloud Monitoring and Cloud Logging.
• Troubleshoot data pipeline issues and implement robust error handling mechanisms.
• Maintain data lineage documentation and impact analysis capabilities.

What you'll need

• 2+ years hands-on experience with BigQuery, Cloud Dataflow, Cloud Composer, Cloud Storage, Cloud Pub/Sub, Dataform/dbt, and Cloud Functions.
• Proficiency in Python/Java and SQL.
• Experience with Spark/Beam development.
• Strong understanding of Apache Beam, Apache Airflow, and distributed computing concepts.
• Experience with both RDBMS (Cloud SQL, PostgreSQL, MySQL) and NoSQL (Bigtable, Firestore) databases.
• Experience with Terraform, Cloud Deployment Manager, or similar tools.
• Bachelor's degree in computer science, Engineering, or related field.
• 2+ years of experience in data engineering or related roles.
• 2+ years of specific experience with Google Cloud Platform.
• Experience with version control systems (Git) and CI/CD pipelines.
• Knowledge of data modelling techniques and dimensional modelling.

Nice to have

• Google Cloud Professional Data Engineer certification.
• Experience with containerization (Docker, Kubernetes/GKE).
• Experience with data governance and compliance frameworks (GDPR, HIPAA, SOX).
• Familiarity with business intelligence tools (Looker, Tableau, Power BI).
• Experience with streaming technologies beyond GCP (Kafka, Spark Streaming).

Details

• Location: Bangalore Office BLS2.