Azure Data Engineer

Job Description

The Azure Data Engineer will be responsible for designing, developing, and maintaining data pipelines, data models, and data storage solutions using Azure services. The candidate will work closely with cross-functional teams to ensure successful delivery of projects and support business objectives. The ideal candidate should have a strong technical background in managing data platforms and architecture using Azure services and experience in data warehousing, data integration, and ETL tools.

Responsibilities:

· Design, develop, and maintain data pipelines using Azure Data Factory to ingest data from various sources such as SQL Server, MongoDB, and Blob Storage into Azure SQL Database and Azure Cosmos DB and Kafka.

· Build data transformation logic using Databricks and Spark for data processing and analytics. Experience in tuning Databricks environments.

· Design / implement data models for data warehousing on Azure SQL DB.

· Configure and manage Hadoop and Spark clusters in Azure HDInsight for big data processing.

· Create and manage data storage solutions using Azure Blob and Data Lake.

· Implement data governance and security measures to protect sensitive data.

· Collaborate with cross-functional teams to ensure successful delivery of projects and support business objectives.

· Mentor junior data engineers and developers and stay up-to-date with emerging technologies and trends in Azure data engineering.

Qualifications:

· BS or MS degree in Computer Science or related fields

· Minimum of 3 years of experience in managing data platforms and architecture using Azure services such as Azure Data Factory, Azure SQL Database, Azure Cosmos DB, and Azure HDInsight.

· Strong experience in data warehousing, data integration, and ETL tools.

· Proficient in programming languages such as Python, SQL, and C#.

· Familiarity with NoSQL databases such as MongoDB and Cassandra.

· Experience in version control tools such as Git and Bitbucket.

· Excellent problem-solving and analytical skills.

· Strong communication and collaboration skills.

· Ability to work independently and in a team environment.

· Desirable: familiarity with Aveva/OSISoft PI system and SCADA system