Senior Data Engineer - AI & Data Platform (Offshore)

Company DescriptionDataFirst Solutions is a leading Data, Analytics, AI, and Digital Transformation company that helps organizations unlock the full value of their data through modern cloud platforms, AI-driven solutions, and intelligent analytics.We enable enterprises to accelerate digital transformation by combining the right people, data, AI technologies, and business expertise to improve operational efficiency, automate decision-making, and create exceptional customer experiences.Our expertise spans Microsoft Fabric, Azure, Databricks, Power BI, AI & Machine Learning, Generative AI, and enterprise data platforms across industries including Hospitality, Automotive, Retail, Telecommunications, Government, Healthcare, Financial Services, and Education.Headquartered in Dubai, DataFirst Solutions is a member of Dubai SME and delivers innovative solutions across the Middle East, Europe, and Asia.Role DescriptionThe Senior Data Engineer – AI & Data Platform is a full-time role responsible for designing, developing, and optimizing modern cloud-native data platforms that power Analytics, Business Intelligence, Machine Learning, and Generative AI solutions.The successful candidate will architect scalable data pipelines, build enterprise-grade Lakehouse solutions, implement semantic data models, and enable AI-ready data foundations using Microsoft Fabric, Azure, Databricks, and other cloud technologies.This role involves close collaboration with Data Scientists, AI Engineers, Solution Architects, Business Analysts, and business stakeholders to deliver high-quality, governed, and scalable data solutions supporting advanced analytics, AI copilots, and enterprise decision-making.The Senior Data Engineer will also contribute to technical architecture, performance optimization, CI/CD implementation, data governance, automation, and best practices across multiple customer engagements.Key ResponsibilitiesDevelop robust batch and real-time data ingestion pipelines from structured, semi-structured, and unstructured data sources.Design and implement scalable enterprise data platforms using Microsoft Fabric, Azure, Databricks, Synapse, or similar cloud technologies.Build and optimize Medallion Architecture (Bronze, Silver, Gold) and Lakehouse solutions.Develop enterprise-grade ETL/ELT pipelines using Azure Data Factory, Microsoft Fabric Data Factory, Databricks, Spark, or equivalent technologies.Design dimensional models, semantic models, and modern analytical data models for enterprise reporting.Build scalable Delta Lake architectures with optimized partitioning, indexing, and performance tuning.Implement data quality validation, monitoring, observability, lineage, and governance frameworks.Collaborate with AI Engineers to prepare feature stores and AI-ready datasets for Machine Learning and Generative AI applications.Develop Retrieval-Augmented Generation (RAG) data pipelines supporting enterprise AI applications.Integrate Vector Databases, Embedding models, and enterprise document repositories for LLM-based solutions.Work with Azure AI Foundry, Azure OpenAI, Microsoft Copilot Studio, Azure AI Search, or equivalent AI services.Implement secure data platforms following enterprise governance, RBAC, Purview, data masking, encryption, and compliance standards.Build CI/CD pipelines using Azure DevOps or GitHub Actions for automated deployment of data solutions.Optimize Spark workloads, SQL performance, storage architecture, and query execution.Mentor junior engineers and contribute to technical standards, reusable frameworks, and engineering best practices.Required QualificationsCandidates should possess strong experience in modern Data Engineering, Data Platform Architecture, and Cloud Analytics solutions.Skills & ExperienceStrong expertise in Microsoft Fabric.7+ years of experience in Data Engineering.Strong expertise in Azure Data Engineering services.Hands-on experience with Azure Data Factory (ADF).Strong experience with Azure Synapse Analytics.Hands-on experience with Azure Data Lake Storage Gen2.Strong experience with Databricks.Expertise in Apache Spark (PySpark preferred).Strong SQL development and query optimization skills.Advanced Python programming.Experience in Delta Lake and Lakehouse Architecture.Strong understanding of Medallion Architecture.Experience in dimensional modeling and semantic data models.Experience with Microsoft Fabric OneLake.Knowledge of Power BI semantic models and Direct Lake architecture.Experience implementing CI/CD using Azure DevOps or GitHub.Experience working with REST APIs and enterprise integrations.Strong knowledge of data governance, metadata management, lineage, and Microsoft Purview.Experience implementing enterprise security, RBAC, Key Vault, and managed identities.Experience building AI-ready enterprise data platforms and preparing datasets for Machine Learning and Generative AI workloads.Hands-on experience integrating Azure OpenAI Services and implementing Retrieval-Augmented Generation (RAG) pipelines using Azure AI Search or similar enterprise search technologies.Strong understanding of Large Language Models (LLMs), Vector Databases, AI embeddings, and Vector Search concepts for enterprise AI applications.Soft Skills:Excellent analytical and problem-solving skills.Strong communication and stakeholder management abilities.Ability to lead technical discussions and mentor engineering teams.Experience working in Agile/Scrum delivery environments.Ability to translate business requirements into scalable technical solutions.Strong ownership, accountability, and attention to detail.EducationBachelor's or Master's degree in Computer Science, Information Technology, Engineering, Data Science, Artificial Intelligence, or a related discipline.Preferred CertificationMicrosoft Certified: Azure Data Engineer Associate (DP-203)Microsoft Certified: Fabric Analytics Engineer Associate (DP-600)Microsoft Certified: Azure AI Engineer Associate (AI-102)Microsoft Certified: Azure Fundamentals (AZ-900)Databricks Certified Data Engineer ProfessionalDatabricks Lakehouse FundamentalsMicrosoft Power BI Data Analyst (PL-300)Candidate Preferred Location: Bangalore, India