Azure +Pyspark(NCS/Job/ 1024)

Data Engineering & ETL:

• Design and develop scalable ETL pipelines using PySpark.

• Optimize Spark jobs for performance and cost-efficiency.

• Perform data cleansing, transformation, and validation.

Azure Cloud Services:

• Build and maintain data solutions using Azure Data Factory, Azure Synapse, Azure Databricks, and Azure Data Lake.

• Use Azure Blob Storage or Data Lake Storage Gen2 for raw/staged/curated data zones.

Big Data Handling:

• Work with large datasets using Spark on Azure Databricks or HDInsight.

• Implement best practices for distributed computing and parallel processing.

Data Governance & Security:

• Ensure compliance with data governance policies.

• Implement data encryption, masking, and role-based access using Azure features.

Monitoring & Troubleshooting:

• Use Azure Monitor, Log Analytics, or Spark UI for job monitoring and debugging.

Collaboration:

• Work with Data Scientists, Analysts, and Business stakeholders to understand data needs.

• Communicate data architecture and flow using tools like Power BI, Azure Purview, or Visio.