Platform Engineer (Databricks - Azure)

Job Description:

Overview

We are seeking a talented and motivated Platform Engineer to join our team. As a Platform Engineer, you will play a crucial role in designing, implementing, and maintaining our data platform using Databricks on the Microsoft Azure cloud. You’ll collaborate with cross-functional teams to ensure the scalability, reliability, and performance of our analytics and data solutions.

Responsibilities

•            Architecture and Design:

o            Provide input into the strategic technology direction for our data platform.

o            Architect scalable and efficient data solutions using Azure Databricks.

•            Data Pipelines and Integration:

o            Design, implement, and manage data pipelines.

o            Ensure seamless data integration across various systems.

o            Optimize data processing workflows.

•            Infrastructure Automation:

o            Develop Terraform Infrastructure as Code (IaC) to automate Databricks environment deployments.

o            Create a library of scripts for automating MLOps environments.

•            Performance Monitoring and Tuning:

o            Monitor and analyze database performance.

o            Fine-tune and optimize the data platform.

•            Collaboration and Communication:

o            Work closely with interdisciplinary teams, including engineers, data scientists, and business stakeholders.

o            Communicate effectively to ensure alignment with business goals.

Qualifications

•            Bachelor’s degree in Computer Science, Engineering, or related field.

•            Proven experience as a Platform Engineer or similar role.

•            Strong expertise in Databricks and Microsoft Azure.

•            Experience with Infrastructure as Code (IaC) and automation tools, preferably Terraform.

•            Knowledge of data catalog tools (e.g., Microsoft Purview).

•            Excellent problem-solving skills and ability to work in a collaborative environment.