Databricks Engineer

Databricks Engineer

London,United Kingdom

Contract

Skills

PySpark

Databricks Engineer

Data Lake Storage, Azure SQL Database, Azure Synapse Analytics

Core skills required is in databricks + Azure cloud services , should know how to setup connectivity from data bricks to other Azure services 

Job Description:

We are looking for a talented Databricks Engineer with expertise in PySpark and experience working with the Azure stack. The ideal candidate will be responsible for designing, implementing, and maintaining robust and scalable data pipelines, ensuring seamless data integration and transformation. Additionally, the candidate will set up and manage connectivity between Databricks and various other services.

Key Responsibilities:

Design and Develop: Create highly performant data ingestion pipelines using Azure Databricks and PySpark.

Data Integration: Integrate, transform, and consolidate data from various structured and semi-structured data sources.

Optimization: Optimize data processing and storage solutions on Azure.

Connectivity Setup: Establish and manage connectivity between Databricks and other Azure services (e.g., Azure Data Lake Storage, Azure SQL Database, Azure Synapse Analytics).

Collaboration: Work closely with data engineers, analysts, and cross-functional teams to deliver quality data products.

CI/CD Pipelines: Implement and manage CI/CD pipelines for data engineering projects using Azure DevOps.

Troubleshooting: Identify and resolve performance issues in data pipelines.

Required Qualifications:

Experience: Minimum of 4 years in data engineering, with at least 2 years of hands-on experience with Azure Databricks and PySpark.

Azure Services: Proficiency in Azure Data Factory, Azure Data Lake Storage, Azure SQL, and Azure Synapse Analytics.

Programming: Strong skills in PySpark and SQL.

Connectivity: Experience in setting up and managing connectivity between Databricks and other services.

Problem-Solving: Excellent analytical and problem-solving abilities.

Communication: Strong communication and collaboration skills.

Preferred Qualifications:

Big Data Technologies: Experience with other big data technologies like Hadoop, Spark, or Kafka.

Cloud Platforms: Familiarity with other cloud platforms such as AWS or GCP.

Certifications: Relevant Azure certifications are a plus.