Senior ETL Developer

Responsibilities:

1. Azure Orchestration and Hosting:

- Implement Azure Logic App flows to seamlessly ingest data from clients' systems into the cloud data store server.
- Configure Azure Functions using NodeJS and SQL to execute key functions, including extracting data from APIs and data objects, transforming it into predefined schemas, and efficiently loading each dataset into the cloud data store.

2. Python-Based ETL Pipeline:

- Implement data processing functions with NodeJS in our ETL pipeline, extracting data from a range of sources and formats and efficiently inserting it into SQL databases.
- Parameterize the ETL pipeline to improve data processing latency and significantly reduce man-hours.

3. Apache Spark Development and Maintenance Docker Deployment:

- Design and implement ETL processes using Apache Spark in standalone mode for enhanced data processing capabilities.
- Deploy and manage ETL workflows in Docker containers, ensuring scalability, flexibility, and ease of deployment.

Required Skills and Qualifications:

1. Education: Bachelor’s degree in computer science, Information Technology, or a related field.

2. Experience :

- Proven 5+ years of experience in implementing ETL processes within cloud environments, particularly Azure.
- Hands-on experience with Azure Logic Apps, Azure Functions, and NodeJS.
- Strong proficiency in Python, with experience in developing Python-based ETL pipelines.
- Experience with Apache Spark and PySpark.
- Experience with Docker for deployment and Git CICD pipelines.
- Comfortable with Windows and Linux Operating systems.

3. Collaboration and Communication:

- Excellent communication and interpersonal skills.