Architect, design, develop and engineering end-to-end data pipelines across multiple data sources and systems of record.
Ensure data quality, integrity, security, and completeness throughout the data lifecycle.
Develop, design data models, data structures and ETL jobs for data acquisition and transformation purposes.
Develop deep understanding of the data sources, implement data standards, maintain data quality and master data management.
Developing data services and API
Work closely with the cloud service providers to ensure completeness and alignment with the service offerings.
Manage and maintain cloud-based data and analytics platform.
Deep understanding of the cloud offerings and engage in quick proof of concepts and proof of value in prototyping data and analytics solutions and derive viability.
Understanding of Unified Data governance and Data Quality Management solutions for managing and governing enterprise data.
Ability to interact with the business stakeholders to understand requirements and translating into technology solutions.
Qualifications
Bachelor’s degree in computer science, computer engineering, or other related field of study with 10+ years of relevant experience
Experience in Azure Cloud platform service offerings
Expertise in Azure SQL with strong knowledge on writing complex SQL queries, Stored procedures, Tuning and Query optimizations.
Experience with Azure API Management Service (APIMS) and developing Azure Functions for exposing API / RESTful data services.
Experience working on real-time data capture, processing, transformation and storing using technologies like Azure Event Hubs, Azure Service Bus, and Azure Stream Analytics
Knowledge of creating pipelines and complex data transformations and manipulations using one of the languages .Net/C#, Python, Java, R, or Scala with Databricks/Spark
Experience in NoSQL Databases and Big data technologies including Azure CosmosDB or other Document stores.
Experience with Azure Purview for automating and managing Data catalog, classification, lineage, and metadata management across enterprise data.
Experience with configuring rules, exceptions and approvals workflows for Data quality management using Profisee.
Experience working with different data storage options including ADSLS Gen2 and Azure BLOB storage.
Understanding of different data formats including Parquet, Avro, CSV, ORC etc.
Understanding of MPP databases and maintain large amount of data processing is a plus.
Experience in creating, scheduling, and monitoring Azure Data Factory (ADF) pipelines.
Experience with Azure Synapse Analytics and Azure Analys Services, Power BI and business Intelligence technologies is a big plus but not mandatory.
Past working experience on a fast paced and agile environment
Experience working with an on-shore / off-shore model.
Build and meet project timelines and manage delivery commitments with proper communication to management.
Collaborate and communicate with key business lines, technology partners, vendors, and architects