Description

Role: Data Engineer Databricks & ADF

Technical Skills:

Summary: The Databricks Engineer is responsible for designing, developing, and maintaining data pipelines and data lakes using Databricks. The ideal candidate will have the following experience :

Notebook configuration in Databricks

Extensive experience writing notebooks, clusters, PySpark coding

Strong experience in Azure data factory, Azure SQL, Synapse, Spark/Databricks

Strong coding skills in Python/Scala/Java

Knowledge on Using Databricks to transform and process data for downstream applications

Big data and modern data lake experience

Good Knowledge on Data Warehousing concepts

Responsibilities:

 

Design, develop, and maintain data pipelines and data lakes using Databricks

Work with other data engineers, data scientists, and business analysts to collect, clean, and transform data

Develop and implement data quality checks and monitoring systems

Work with stakeholders to understand their data needs and develop solutions

Stay up-to-date on the latest data engineering technologies and best practices

 

Secondary Skill:

 

Experience with Alation

Experience connecting Power BI and Azure Databricks datasets

Knowledge of Power BI or Tableau

Azure certification is a plus

Insurance industry knowledge

 

Soft skills:

 

Excellent written and verbal communication, intellectual curiosity, a passion to understand and solve problems, consulting & customer service

Structured and conceptual mindset coupled with strong quantitative and analytical problem-solving aptitude

Ability to work independently in an Agile development environment

Exceptional interpersonal and collaboration skills within a team environment

 

Qualifications:

 

Bachelor's degree in computer science, data science, or a related field

5+ years of experience in data engineering

Experience with notebook configuration, reading from data loads, using Databricks to transform and process downstream, Elation, and Power BI

Experience with cloud computing platforms (AWS/Azure/GCP) "


 

Key Skills
Education

Bachelor's Degree

Back To Jobs