Senior Azure Data Engineer

Job Description

​
 

Responsibilities
 

• Architect, design, develop and engineering end-to-end data pipelines across multiple data sources and systems of record.
   
• Ensure data quality, integrity, security, and completeness throughout the data lifecycle.
   
• Develop, design data models, data structures and ETL jobs for data acquisition and transformation purposes.
   
• Develop deep understanding of the data sources, implement data standards, maintain data quality and master data management.
   
• Developing data services and API
   
• Work closely with the cloud service providers to ensure completeness and alignment with the service offerings.
   
• Manage and maintain cloud-based data and analytics platform.
   
• Deep understanding of the cloud offerings and engage in quick proof of concepts and proof of value in prototyping data and analytics solutions and derive viability.
   
• Understanding of Unified Data governance and Data Quality Management solutions for managing and governing enterprise data.
   
• Ability to interact with the business stakeholders to understand requirements and translating into technology solutions.
   

Qualifications
 

• Bachelor’s degree in computer science, computer engineering, or other related field of study with 10+ years of relevant experience
   
• Experience in Azure Cloud platform service offerings
   
• Expertise in Azure SQL with strong knowledge on writing complex SQL queries, Stored procedures, Tuning and Query optimizations.
   
• Experience with Azure API Management Service (APIMS) and developing Azure Functions for exposing API / RESTful data services.
   
• Experience working on real-time data capture, processing, transformation and storing using technologies like Azure Event Hubs, Azure Service Bus, and Azure Stream Analytics
   
• Knowledge of creating pipelines and complex data transformations and manipulations using one of the languages .Net/C#, Python, Java, R, or Scala with Databricks/Spark
   
• Experience in NoSQL Databases and Big data technologies including Azure CosmosDB or other Document stores.
   
• Experience with Azure Purview for automating and managing Data catalog, classification, lineage, and metadata management across enterprise data.
   
• Experience with configuring rules, exceptions and approvals workflows for Data quality management using Profisee.
   
• Experience working with different data storage options including ADSLS Gen2 and Azure BLOB storage.
   
• Understanding of different data formats including Parquet, Avro, CSV, ORC etc.
   
• Understanding of MPP databases and maintain large amount of data processing is a plus.
   
• Experience in creating, scheduling, and monitoring Azure Data Factory (ADF) pipelines.
   
• Experience with Azure Synapse Analytics and Azure Analys Services, Power BI and business Intelligence technologies is a big plus but not mandatory.
   
• Past working experience on a fast paced and agile environment
   
• Experience working with an on-shore / off-shore model.
   
• Build and meet project timelines and manage delivery commitments with proper communication to management.
   
• Collaborate and communicate with key business lines, technology partners, vendors, and architects