Lead Data Engineer

We have an immediate long-term opportunity with one of our prime clients for a position Lead Data Engineer to work on Remote basis.

Job Description:

We are seeking a highly skilled and experienced Business Intelligence Data Engineer to join our team. The ideal candidate will have a strong background in data engineering, with specific expertise in Power BI, SQL, Python, and Databricks. This role will focus on designing and developing data models, ensuring data quality, and implementing robust data solutions.

Key Responsibilities:

• Design, develop, and maintain scalable data pipelines and systems using Databricks, Python, and SQL.
• Implement and optimize data models, focusing on Kimball dimensional modeling techniques.
• Develop and manage Slowly Changing Dimensions (SCD) and Snapshot Tables to ensure accurate historical data tracking.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver high-quality data solutions.
• Ensure data quality and integrity through rigorous testing and validation processes.
• Monitor and troubleshoot data pipeline performance, ensuring efficient data flow and processing.
• Stay updated with the latest industry trends and best practices in data engineering and analytics.
• Ensure data quality and reliability using Python libraries, with a preference for experience in Great Expectations.

Qualifications:

• Over 7 years of experience in SQL, with a proven track record of developing and optimizing complex queries.
• Prior experience with Power BI, including creating and managing dashboards and reports.
• Extensive experience with Kimball dimensional modeling, including the implementation of Slowly Changing Dimensions (SCD) and Snapshot Tables.
• Proficiency in Databricks, with hands-on experience in developing data models and working with notebooks in SQL and Python.
• Knowledge of Python libraries, with experience in Great Expectations being a bonus.
• Excellent problem-solving skills and the ability to work independently and as part of a team.
• Strong communication skills, with the ability to convey complex technical concepts to non-technical stakeholders.