Azure Data Engineer

Azure Data Platform Engineer – Grade 7

The Azure Data Platform Engineer is responsible for implementation, designing, developing, debugging, integrating, transforming, consolidating, and troubleshooting structured and unstructured data systems for analytic and relational solutions. They will assist stakeholders, and team members to understand data through exploration, and strategy. Their main responsibilities include building out data processing pipelines, administering, managing, and building out structural and nonstructural Azure data platform solutions, applying and implementing secure data principles and patterns, creating advanced and optimized queries.  They will work on many platforms such as Azure SQL, Cosmos DB, Azure Data Factory, Azure Synapse. This role will work closely with Solution Architects, Software Engineers, DevOps Engineers, QA Engineers, Project Managers, Information Security Engineers, and Delivery Managers.

What You Will Be Doing

Tech Breakdown (Coding and Collaboration)

• 30% Data Exploration and Design (Security, Research, Quality, Performance, Design)
• 30% Data Pipeline Development (Secure Development, Implementation, Unit Testing)
• 20% Azure Data Management (Provisioning, Authentication, Authorization, Monitoring)
• 20% Software Deployment (DevSecOps, Release management)

Qualifications And Requirements:

• Must be a team player with get it done attitude
• Design and architect end-to-end data platform solutions on the Azure cloud platform, considering scalability, reliability, performance, and security.
• Collaborate with stakeholders to understand business requirements and translate them into technical specifications and data architecture designs.
• Develop and implement data models, schemas, and database structures to ensure efficient data storage and retrieval.
• Design and implement data integration solutions, including data ingestion, transformation, and data pipeline orchestration using Azure data services (e.g., Azure Data Factory, Azure Databricks, Azure Logic Apps, Azure Functions).
• Optimize and tune data solutions for performance, ensuring efficient query execution and data processing.
• Implement data security and compliance measures, including data encryption, access controls, and data masking techniques.
• Work closely with data analysts, data scientists, and other stakeholders to understand their data requirements and provide guidance on data availability and accessibility.
• Collaborate with infrastructure teams to design and implement robust data storage and backup strategies, leveraging Azure storage services.
• Monitor and troubleshoot data platform solutions to ensure high availability, reliability, and performance.
• Stay up-to-date with the latest trends and advancements in Azure data services, data engineering, and data architecture, and provide recommendations for adopting new technologies and best practices.
• Eager to share knowledge with teammates in all forums and eager to learn from them
• Bachelor’s degree in computer science/engineering or related technical field or equivalent experience
• Comprehensive knowledge of full Software Development Lifecycle (SDLC) including Continuous Integration Continuous Development (CI/CD) models.
• Strong knowledge of Data Structures and Algorithms.
• 8+ years of Software development with relational databases. Preferably .Net - C# - SQL
• Demonstrable debugging skills in MS SQL desired.
• Demonstrable experience with Relational and Non-relational databases.
• Demonstrable experience building data pipelines, transformations, analytical environments
• Demonstrable experience with Azure Data Platform (Synapse, SQL, Cosmos DB, Databricks, Machine Learning workspace)
• Collaborate with Product Managers, Solution Architects, DevOps and Information Security teams to understand data requirements for software development in an Agile environment
• Compose and maintain detailed design specifications and component documents.
• Design alongside a team of engineers focusing on 'data first' thinking while incorporating different end user experiences.
• Embrace and follow best practices and coding standards (e.g., code reviews, logging and instrumentation, static/dynamic code analysis, code coverage, unit, integration tests).
• Strong initiative to find ways to improve solutions, systems, and processes

Must  have skills:

• Azure Databricks
• Big Data concepts
• Data Lake concept
• Machine Learning concepts

 1 or more certifications:

• Azure Data Engineer Associate
• Azure Cosmos DB Specialty
• Azure Administrator Associate
• Azure Data Scientist Associate
• Azure DevOps Engineer Expert