AI Fabric Data Engineer (RARR Job 6407)

Role Summary

We are seeking an experienced Microsoft Fabric Platform Engineer / Architect to design, build, and manage enterprise-scale data platforms using the Microsoft Fabric ecosystem. The ideal candidate will have expertise in data engineering, platform architecture, semantic modeling, security, governance, and AI integration to deliver scalable, secure, and high-performance analytics solutions.

Key Responsibilities

Microsoft Fabric Platform Architecture

• Design, build, and maintain end-to-end data solutions across the Microsoft Fabric ecosystem, including Lakehouses, Warehouses, Data Pipelines, Dataflows Gen2, Eventstreams, and Real-Time Intelligence.

• Design and implement Medallion Architecture (Bronze, Silver, Gold) and Data Mesh patterns aligned with enterprise governance standards.

• Manage Microsoft Fabric workspaces, capacity planning, domain administration, and deployment pipelines across Development, Test, and Production environments.

• Optimize Delta Lake performance using techniques such as Z-Ordering, OPTIMIZE, VACUUM, partition pruning, and performance tuning.

• Develop and maintain Spark notebooks using PySpark or Scala, along with T-SQL stored procedures.

Data Engineering & Integration

• Design and build scalable data ingestion and transformation pipelines.

• Develop reusable ETL/ELT frameworks and automate data workflows.

• Implement incremental loading, monitoring, error handling, and data validation.

• Integrate enterprise data from multiple source systems into Microsoft Fabric.

• Experience with Azure Data Factory (ADF) for orchestration and migration is preferred.

Semantic Modeling & Power BI

• Build and manage enterprise-grade Direct Lake semantic models within Microsoft Fabric.

• Develop advanced DAX measures, calculation groups, field parameters, and dynamic format strings.

• Design Star Schema and Snowflake Schema models optimized for Direct Lake performance.

• Implement aggregation strategies, composite models, and semantic model optimization.

• Establish CI/CD for semantic models using XMLA Endpoints, Tabular Editor 3, Git-connected Fabric workspaces, and deployment pipelines.

• Define enterprise standards for naming conventions, measure organization, and model documentation.

Data Security & Governance

• Implement Row-Level Security (RLS), including static, dynamic, user-based, and manager hierarchy access models.

• Configure Object-Level Security (OLS) to secure sensitive tables and semantic model objects.

• Design and implement Column-Level Security (CLS) within Fabric Warehouse and Lakehouse SQL endpoints.

• Apply Dynamic Data Masking (DDM) to protect sensitive and regulated data.

• Manage security using Microsoft Entra ID (Azure Active Directory) group-based access controls.

• Prepare security documentation and compliance artifacts for governance and audit requirements.

AI & Automation

• Design and develop Microsoft Fabric Agents using the Fabric Agent Framework.

• Integrate Fabric Agents with Microsoft Copilot Studio to enable conversational analytics and intelligent business workflows.

• Integrate Microsoft Fabric with external AI/LLM platforms (such as Claude or OpenAI) for data enrichment, classification, summarization, and intelligent automation.

• Develop Power Automate workflows and Azure Logic Apps to trigger Fabric Data Pipelines, Agents, and Eventstreams from Microsoft 365 applications.

• Evaluate and implement emerging Microsoft Fabric AI capabilities, including AutoML, AI Skills, and OneLake AI Search.

Governance & Quality

• Establish enterprise governance standards for Fabric workspaces, endorsements, sensitivity labels, lineage tracking, and Microsoft Purview integration.

• Define and implement data quality frameworks using Great Expectations, dbt, or native Microsoft Fabric validation capabilities.

• Prepare architecture documents, Architecture Decision Records (ADRs), data dictionaries, semantic model documentation, and operational runbooks.

• Conduct code reviews, mentor engineering teams, and enforce development best practices.

• Collaborate with business stakeholders, architects, and cross-functional teams to deliver scalable and enterprise-grade Microsoft Fabric solutions.

Required Skills

• Microsoft Fabric

• Lakehouse & Data Warehouse

• Data Pipelines

• Dataflows Gen2

• Eventstreams

• OneLake

• Real-Time Intelligence

• PySpark / Spark

• Scala (preferred)

• T-SQL

• Delta Lake

• Azure Data Factory (ADF)

• Power BI

• Direct Lake

• DAX

• Star & Snowflake Schema

• Microsoft Entra ID

• RLS, OLS, CLS

• Dynamic Data Masking (DDM)

• Microsoft Purview

• Git & CI/CD

• XMLA Endpoints

• Tabular Editor 3

• Power Automate

• Azure Logic Apps

• Microsoft Copilot Studio

• Microsoft Fabric Agents

• AI/LLM Integration (Claude, OpenAI, or similar)

Preferred Experience

• 8–15 years of experience in Data Engineering, Analytics, or Data Platform development.

• 2+ years of hands-on Microsoft Fabric experience.

• Strong experience with Azure Data Platform technologies.

• Proven expertise in designing enterprise-scale analytics platforms and modern data architectures.

• Experience leading technical teams, mentoring engineers, and driving architecture decisions is highly desirable.