Senior Big Data Engineer

Job Description:
Responsibilities:
- Design, develop, and implement scalable data pipelines and ETL processes using Java, Python, and Spark.
- Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and design efficient solutions.
- Manage and optimize Spark clusters to ensure high performance and reliability.
- Perform data exploration, data cleaning, and data transformation tasks to prepare data for analysis and modeling.
- Develop and maintain data models and schemas to support data integration and analysis.
- Implement data quality and validation checks to ensure accuracy and consistency of data.
- Utilize REST API development skills to create and integrate data services and endpoints for seamless data access and consumption.
- Monitor and troubleshoot data pipeline performance, identifying and resolving bottlenecks and issues.
- Stay updated with the latest technologies and trends in big data, data engineering, data science, and REST API development, and provide recommendations for process improvements.
- Mentor and guide junior team members, providing technical leadership and sharing best practices.

Qualifications:
- Master's degree in Computer Science, Data Science, or a related field.
- Minimum of 3 years of professional experience in data engineering, working with Java, Python, Spark, and big data technologies.
- Strong programming skills in Java and Python, with expertise in building scalable and maintainable code.
- Proven experience in Spark cluster management, optimization, and performance tuning.
- Solid understanding of data science concepts and experience working with data scientists and analysts.
- Proficiency in SQL and experience with relational databases (e.g., Snowflake, Delta Tables).
- Experience in designing and developing REST APIs using frameworks such as Flask or Spring.
- Familiarity with cloud-based data platforms (e.g.Azure)
- Experience with data warehousing concepts and tools (e.g., Snowflake, BigQuery) is a plus.
- Strong problem-solving and analytical skills, with the ability to tackle complex data engineering challenges.
- Excellent communication and collaboration skills, with the ability to work effectively in a team-oriented environment.