Description

Job Overview

 

We seek a highly skilled Data Engineer with extensive expertise in Azure DevOps to join our dynamic team. The ideal candidate will have 10-13 years of experience in data engineering, cloud architecture, and DevOps practices, specializing in the Microsoft Azure ecosystem. The candidate will be responsible for designing, building, and maintaining scalable data pipelines and infrastructure using Azure services while implementing robust CI/CD practices to streamline deployment and improve operational efficiency.

 

Responsibilities

Design, develop, and maintain large-scale data processing systems and pipelines using Azure Data Services (Data Factory, Data Lake, Synapse, Databricks) and big data technologies (Spark/Kafka).

  • Lead the implementation of Azure DevOps CI/CD pipelines for automated build, test, and deployment processes, ensuring data integrity, scalability, and performance.
  • Collaborate with cross-functional teams to architect, build, and optimize Azure-based data platforms for large volumes of structured and unstructured data.
  • Manage data ingestion, transformation, and orchestration pipelines using ETL/ELT processes and Azure-native services (Storage, Blob Storage, Databricks).
  • Implement Infrastructure as Code (IaC) with Terraform or ARM templates to automate and manage cloud infrastructure.
  • Ensure data security, encryption, and compliance (GDPR, HIPAA) in collaboration with the security team, adhering to Azure Security Center protocols.
  • Troubleshoot and optimize data workflows using Azure Monitor, Log Analytics, and Application Insights for performance, reliability, and scalability.

Develop and maintain data models and database designs aligned with business needs for optimal performance and scalability.

 

Skills & Qualifications

  • 10-13 years of experience in data engineering, cloud architecture, and DevOps, with a focus on Azure technologies.
  • Expertise in Azure DevOps, including CI/CD pipeline development, version control with Git, and automation using Azure Pipelines and Terraform/ARM templates.
  • Should have experience in writing PowerShell scripts for automation.
  • Hands-on experience with Azure Data Services: Azure Data Lake, Azure SQL, Azure Synapse Analytics, Azure Databricks, and Azure Data Factory.
  • Strong proficiency in programming languages such as Python, PySpark / SparkSQL, Java or Scala, and SQL.
  • Experience with big data tools (e.g., Spark, Hive, Kafka) and real-time data processing.
  • Proven track record of managing data migrations, transformations, and orchestrations in large-scale environments.
  • Experience with cloud architecture and hybrid cloud scenarios, with a focus on performance tuning, cost optimization, and scalability.
  • Familiarity with database technologies such as SQL Server, AzureSQL, or MySQL database
  • Knowledge of data security and compliance standards (GDPR, HIPAA) and experience implementing encryption and access control protocols.

Strong experience in Agile development practices and collaboration tools like Jira and Confluence.

 

Nice to Have

  • Certifications in Azure DevOps, Azure Data Engineer, or related cloud technologies.

Familiarity with other cloud platforms like AWS or Google Cloud.


 

Key Skills
Back To Jobs