Senior AWS Data Engineer / Lead

Join us as a Senior AWS Data Engineer / Lead and spearhead our data initiatives and leverage your expertise in the AWS cloud to design, build, and maintain data-driven solutions.

Job Description:

• Orchestration of Amazon MWAA: Expertise in managing Amazon Managed Workflows for Apache Airflow (MWAA) for data orchestration, including ingestion, transformation, and data integrity tasks.
• Transition to ELT: Implementing ELT (Extract, Load, Transform) approach aligned with Gartner’s data transformation strategy, focusing on metadata-driven integration design and rapid onboarding of new systems.
• DataOps Delivery: Adopting a DataOps approach with Informatica, pySpark, Python shell, and SQL programming skills to build data pipelines, along with CI/CD pipelines using GitHub Actions and Infrastructure as Code (IaC) with Terraform.
• Data Lake House Architecture: Utilizing AWS services like S3 and Amazon Redshift to create a true Data Lake house architecture with native Role-Based Access Control (RBAC) and security controls, integrating Snowflake for enhanced RBAC and observability using AWS CloudWatch and Apache Airflow.
• Event-Driven Architecture: Implementing event-based architecture with AWS EventBridge to monitor and audit AWS environments, enabling real-time response to operational changes and preventing infrastructure vulnerabilities.

Skills and Experience Required:

• Bachelor’s or Master’s degree in Computer Science or related fields with 8-10 years of professional experience.
• Proficiency in managing Amazon MWAA for Apache Airflow orchestration.
• Experience transitioning from ETL to ELT approach with a focus on metadata-driven integration.
• Strong programming skills in pySpark, Python shell, and SQL for building data pipelines.
• Hands-on experience with Informatica, GitHub Actions for CI/CD pipelines, and Terraform for Infrastructure as Code.
• Expertise in implementing Data Lake house architecture using AWS services like S3, Amazon Redshift, and Snowflake.
• Familiarity with Role-Based Access Control (RBAC), observability using AWS CloudWatch and Apache Airflow, and event-based architecture with AWS EventBridge.