Data Engineer(NCS/Job/ 770)

Key Responsibilities:

• Design, develop, and maintain scalable data pipelines using Scala/Python (PySpark).
• Work with big data technologies to process and manage large datasets efficiently.
• Develop and optimize ETL workflows using Apache Airflow.
• Implement real-time and batch data processing solutions using Kafka and other streaming technologies.
• Build and maintain distributed databases (SQL & NoSQL) for efficient data storage and retrieval.
• Develop and manage CI/CD pipelines for data engineering workflows.
• Work with Google Cloud Platform (GCP) Data Tech Stack, including BigQuery, Dataflow, Dataproc, Pub/Sub, and Cloud Storage.
• Implement data governance, monitoring, and security best practices.
• Collaborate with data scientists, analysts, and software engineers to provide reliable data solutions.
• Optimize query performance and troubleshoot issues related to data pipelines and infrastructure.

Required Skills & Qualifications:

• Strong programming skills in Scala and/or Python (PySpark).
• Hands-on experience with big data technologies such as Hadoop, Spark, or Databricks.
• Experience in scripting (Shell/Python) for automation.
• Good understanding of GCP Data Tech Stack (BigQuery, Dataflow, Dataproc, Pub/Sub, Cloud Storage).
• Experience with Apache Airflow for orchestrating data pipelines.
• Familiarity with CI/CD pipelines and DevOps practices.
• Experience with Kafka for real-time data streaming.