Pyspark Databricks (NCS/Job/ 2392)

We are seeking a highly skilled Data Engineer with strong expertise in PySpark, Databricks, and SQL to design, develop, and optimize data pipelines and solutions for large-scale data processing. The ideal candidate will work closely with cross-functional teams to ensure efficient data integration and transformation for analytics and reporting.

Key Responsibilities:

• Design and implement scalable ETL pipelines using PySpark and Databricks.
• Develop and optimize SQL queries for data extraction, transformation, and loading.
• Collaborate with data analysts, data scientists, and business stakeholders to deliver high-quality data solutions.
• Ensure data quality, integrity, and security across all processes.
• Monitor and troubleshoot data workflows to maintain performance and reliability.
• Document technical designs, processes, and best practices.

Mandatory Skills:

• PySpark: Strong experience in distributed data processing and transformations.
• Databricks: Hands-on experience with Databricks platform for big data analytics and pipeline development.
• SQL: Advanced proficiency in writing complex queries, performance tuning, and working with relational databases.

Preferred Skills:

• Experience with Azure Data Lake, AWS, or GCP.
• Knowledge of Delta Lake, Spark Streaming, and data warehousing concepts.
• Familiarity with CI/CD pipelines and version control (Git).
• Understanding of data governance and security best practices.

Qualifications:

• Bachelor’s or Master’s degree in Computer Science, Information Technology, or related field.
• 6+ years of experience in data engineering or big data development.