PySpark Lead

Job Description:
 

Lead a team of data engineers in the design and implementation of big data solutions using PySpark

Collaborate with cross-functional teams to understand business needs and develop data-driven solutions

Design and implement scalable and robust data pipelines using PySpark

Ensure data accuracy and integrity by implementing data validation and quality checks

Mentor and coach team members in PySpark and big data best practices

Stay up-to-date with the latest developments in big data technologies and incorporate them into the team's workflow

Requirements

Requirements:
 

8+ years in Pyspark/ Data engineer development and implementation

Bachelor's or Master's degree in Computer Science, Engineering, or a related field

Proven experience as a PySpark Lead or similar role

Expertise in PySpark and big data technologies such as Hadoop, Hive, and Spark