Data Engineer - Big Data & Hadoop Admin

Key Responsibilities:
Big Data & Hadoop Administration:
Manage and administer Hadoop (Hortonworks) clusters and Cloudera Data Platform (CDP).
Work extensively with Spark, Scala, and PySpark for data processing and ingestion.
Database Management & Migration:
Oversee database migration processes to MS SQL Server and Hadoop.
Tune database queries and optimize performance for OLTP and OLAP systems.
Scripting & Automation:
Develop and maintain scripts using Batch, Shell, and Python.
Set up CI/CD pipelines for database changes using GitHub, Jenkins, and Liquibase.
Technical Support & Collaboration:
Provide support and guidance to team members, ensuring a deep understanding of project requirements.
Work effectively in cross-functional teams and various team sizes.
Additional Responsibilities:
Utilize Agile, Scrum methodologies, and tools like Jira and Git.
Engage in data migration from Hadoop to CDP as needed.
Desirable Skills:
Experience with Agile, Scrum, Jira, Git, SVN, and Liquibase
Familiarity with data migration from Hadoop to CDP
Competencies:
Digital: Big Data and Hadoop Ecosystems
Digital: Microsoft SQL Server 2019
Digital: Jenkins
Digital: PySpark
Experience Required:
6-8 years in a similar role with demonstrated experience in Big Data and Hadoop ecosystems.