Databricks Data Engineer (Job No 941)

Key Responsibilities:

1. Develop & Optimize ETL Pipelines:
   • Build robust and scalable data pipelines using ADF, Databricks, and Python for data ingestion, transformation, and loading.
2. Data Modeling & Systematic Layer Modeling:
   • Design logical, physical, and systematic data models for structured and unstructured data.
3. Integrate SAP IS-Auto:
   • Extract, transform, and load data from SAP IS-Auto into Azure-based data platforms.
4. Database Management:
   • Develop and optimize SQL queries, stored procedures, and indexing strategies to enhance performance.
5. Big Data Processing:
   • Work with Azure Databricks for distributed computing, Spark for large-scale processing, and Delta Lake for optimized storage.
6. Data Quality & Governance:
   • Implement data validation, lineage tracking, and security measures for high-quality, compliant data.
7. Collaboration:
   • Work closely with business analysts, data scientists, and DevOps teams to ensure data availability and usability.
8. Testing and Debugging:
   • Write unit tests and perform debugging to ensure the Implementation is robust and error-free.
   • Conduct performance optimization and security audits.

Required Skills and Qualifications:

1. Azure Cloud Expertise:
   • Strong experience in Azure Data Factory (ADF), Databricks, and Azure Synapse.
2. Programming:
   • Proficiency in Python for data processing, automation, and scripting.
3. SQL & Database Skills:
   • Advanced knowledge of SQL, T-SQL, or PL/SQL for data manipulation.
4. SAP IS-Auto Data Handling:
   • Experience integrating SAP IS-Auto as a data source into data pipelines.
5. Data Modeling:
   • Hands-on experience in dimensional modeling, systematic layer modeling, and entity-relationship modeling.
6. Big Data Frameworks:
   • Strong understanding of Apache Spark, Delta Lake, and distributed computing.
7. Performance Optimization:
   • Expertise in query optimization, indexing, and performance tuning.
8. Data Governance & Security:
   • Knowledge of RBAC, encryption, and data privacy standards.

Preferred Qualifications:

1. Experience with CI/CD for data pipelines using Azure DevOps.
2. Knowledge of Kafka/Event Hub for real-time data processing.
3. Experience with Power BI/Tableau for data visualization (not mandatory but a plus).