GCP Data Engineer(Job No 1063)

• Design, develop, deploy, and maintain scalable, resilient, and secure data pipelines (batch and streaming) on GCP using services like Cloud Dataflow, Cloud Composer, and Cloud Functions.
• Manage and operate core data infrastructure components on GCP, including BigQuery, Bigtable, Google Kubernetes Engine (GKE) for data workloads, and related networking/security configurations.
• Implement and champion DataOps principles, including CI/CD for data pipelines (using tools like Jenkins), automated testing, robust monitoring (Cloud Monitoring, Cloud Logging), and alerting strategies.
• Optimize data storage, processing, and query performance across the GCP data stack, focusing on efficiency and cost management.
• Troubleshoot and resolve complex issues related to data infrastructure, pipeline failures, and data quality.
• Collaborate closely with data scientists, analysts, software engineers, and product managers to understand data requirements and deliver effective solutions.
• Ensure data governance, data quality, and security best practices are implemented and adhered to within the GCP environment.
• Stay current with the latest advancements in GCP data services and broader data engineering/DataOps trends.
• Evaluate and leverage GCP's AI/ML capabilities where applicable, potentially contributing to intelligent monitoring or operational improvements.

Required Qualifications & Skills:

• Bachelor's or master's degree in computer science, Engineering, Information Technology, or a related field.
• 4-7 years of professional experience in data engineering, software engineering with a data focus, or a related infrastructure role.
• Minimum of 4 years of significant hands-on experience designing, building, and operating data solutions specifically on Google Cloud Platform (GCP).
• Proven, in-depth experience with core GCP data services:
  • Data Processing: Cloud Dataflow, Cloud Functions
  • Orchestration: Cloud Composer (or managed Airflow)
  • Storage/Warehousing: BigQuery, Bigtable, Google Cloud Storage (GCS)
  • Operations/Monitoring: Cloud Logging, Cloud Monitoring, Cloud Trace
  • Containerization (for data workloads): Google Kubernetes Engine (GKE)
• Strong experience in managing data infrastructure, including provisioning, configuration, performance tuning, and cost optimization.
• Proficiency in programming languages commonly used in data engineering, particularly Python and SQL.
• Solid understanding of ETL/ELT processes, data modeling, data warehousing concepts, and database design.
• Experience implementing monitoring, logging, and alerting for data pipelines and infrastructure.
• Familiarity with infrastructure-as-code (IaC) tools (e.g., Terraform).
• Experience with CI/CD tools and practices (e.g., Cloud Build, Jenkins, GitLab CI).
• Understanding of containerization (Docker) and orchestration (Kubernetes).
• Awareness of GCP's AI/ML offerings (e.g., Vertex AI, BigQuery ML).
• Excellent problem-solving and troubleshooting skills.
• Strong communication and collaboration abilities.

Preferred Qualifications (Bonus Points):   

• Demonstrable experience in building and deploying AI/ML-driven agents or tools specifically aimed at monitoring, anomaly detection, or automated troubleshooting for data pipelines and infrastructure issues.
• Relevant Google Cloud Professional certifications (e.g., Professional Data Engineer, Professional Cloud Architect).
• Experience with other major cloud platforms (AWS, Azure).
• Experience with streaming technologies (e.g., Pub/Sub, Kafka).
• Experience with data governance platforms or tools.