Data Platform Engineer

Responsibilities:

· Design, develop, and maintain data pipelines using AWS, Apache Airflow to automate the extraction, transformation, and loading (ETL) of data from various sources into Snowflake.
· Collaborate with cross-functional teams to understand data requirements and design scalable and efficient data models and architectures.
· Develop reusable scripts, procedures, and workflows to standardize data processing tasks and ensure consistency and reliability across pipelines.
· Optimize and tune data pipelines for performance, scalability, and cost-effectiveness, leveraging best practices and industry standards.
· Implement monitoring and alerting solutions to proactively identify and address issues in data pipelines, ensuring high availability and reliability.
· Document data engineering processes, procedures, and best practices, and provide training and support to team members as needed.
· Design and implement reusable Directed Acyclic Graphs (DAGs) in Apache Airflow to orchestrate complex workflows and dependencies between tasks within data pipelines.
· Define task dependencies, scheduling intervals, retries, and error handling strategies within DAGs to ensure the reliable execution of data processing tasks.
· Implement dynamic DAG generation and parameterization techniques to support flexible and scalable pipeline configurations.
· Stay current with emerging technologies and industry trends in Airflow & data engineering and analytics, continuously evaluating and incorporating new tools and techniques to improve our data platform offerings.
· Good understanding of various snowflake features like Snowpipes, SnowTasks, Dynamic Data masking, Row access policies, Object tagging, RBAC, Streams etc.
· Design, implement, and maintain CI/CD pipelines using DevOps tools like Terraform, Cloud formation & Jenkins for automated build, test, and deployment processes.
· Create, Manage, and optimize infrastructure on AWS, ensuring high availability, scalability, and cost-effectiveness.
· Build frameworks to automate CI/CD deployments on snowflake.
has context menu