Data Engineer

Why ALOIS AUSTRALIA?

ALOIS is a global consulting, services and staffing solutions partner with dedicated teams to service a wide range of specialties and domains.

Our goal is to empower our employees by creating a support system and process that inspires them to maximize their personal and professional potential.

At ALOIS AUSTRALIA, we are passionate about providing equal employment opportunities and embracing diversity to the benefit of all. We actively encourage applications from any background.

About the job:

Job Description:

As a Data Engineer, you will be responsible for designing, developing, and maintaining robust data infrastructure and pipelines. You will work closely with cross-functional teams to understand business requirements, optimize data workflows, and ensure the reliability and performance of our data systems.

Key Responsibilities:

Design, build, and optimize data processing pipelines using Apache Spark, Python, Scala

Develop and maintain data ingestion and extraction processes, including streaming data pipelines using Kafka and batch processing workflows

Implement performance tuning techniques to optimize data processing and query performance, ensuring scalability and efficiency

Collaborate with DevOps teams to deploy and manage data infrastructure on NetApp S3 (very similar to AWS S3), Kubernetes

Containerize data applications using Docker and orchestrate deployment using Kubernetes for scalability and reliability

Develop and maintain unit tests using frameworks like pytest, junit, to ensure the quality and reliability of data pipelines

Implement and adhere to best practices for data governance, security, and compliance

Utilize Behavior-Driven Development (BDD) tools like Cucumber and Lettuce to write and execute test scenarios for data workflows

Requirements:

Proven experience in designing and building scalable data pipelines using Apache Spark, Python, and Scala

Strong understanding of data warehousing concepts, ETL processes, and data modelling techniques

Experience with performance tuning and optimization of Spark jobs and SQL queries