Azure Data Engineer

Data Engineer (DevOps)
Key Accountabilities:
 Data Integration and Orchestration: Designs and implements data pipelines using
Azure Data Factory to orchestrate data movement from various sources (databases,
APIs, etc.) into Azure storage solutions (e.g., Azure Data Lake Storage, Azure Blob
Storage).
 ETL Processing and Transformation: Develops and manages ETL (Extract, Transform,
Load) processes within Azure Databricks clusters, utilizing Spark (PySpark/Scala) for
efficient data transformation.
 Performance Optimization: Continuously optimizes data pipelines for efficiency and
scalability using Spark techniques within Databricks and optimizes storage solutions
for optimal performance.
 Data Pipeline CI/CD: Designs, implements, maintains and enhanced CI/CD pipelines
for data pipelines using Azure DevOps services (e.g., Azure Pipelines, Azure Repos).
This includes automating testing, deployment, and configuration management for
data infrastructure on Azure.
 Azure Infrastructure Management: Provisions, configures, and manages
infrastructure for data pipelines using Azure services (e.g., Azure Data Factory, Azure
Databricks VMs, Azure Functions).
 Version Control and Collaboration: Utilizes Bitbucket, Azure DevOps services (e.g.,
Azure Repos) to manage code changes for data pipelines and collaborates with other
engineers on infrastructure development.
 Security and Compliance: Ensures secure deployment of data pipelines on Azure,
adhering to data privacy and compliance regulations using Azure Security Center and
other security tools.
 Monitoring and Alerting: Sets up Azure Monitor to track data pipeline health and
performance and implements alerting systems for potential issues through Azure
Alerts.
 Scripting: Writes scripts using Azure Functions/ Azure Services CLI to automate tasks
related to data pipeline deployment and management.
Additional Skills:
 Experience with Azure DevOps, Bitbucket tools and methodologies.
 Familiarity with Infrastructure as Code (IaC) tools like Azure Resource Manager (ARM)
templates or Bicep.
 Understanding of cloud security best practices in Azure.
Qualification:
 Bachelor’s degree in IT, computer science, or a relevant field.
 2-5 years of experience in the data engineering/analyst domain.
 Able to work independently in a fast-paced and complex environment.
 Good communication skills (both verbal and written) – English is mandatory.
 Expert knowledge in data, data engineering and other related subdomains.
 Background in cloud, cloud native applications and data flows.
 Working knowledge of MS SQL, PostgreSQL, CDC, data Streaming Application (e.g.
Apache Kafka, Qlik Replicate) and other related systems.