AWS Redshift Data Engineer

𝐑𝐞𝐬𝐩𝐨𝐧𝐬𝐢𝐛𝐢𝐥𝐢𝐭𝐢𝐞𝐬:

• Collaborate with data engineering, business analysts, and development teams to design, develop, test, and maintain robust and scalable data pipelines from Workday to AWS Redshift.
• Architect, implement, and manage end-to-end data pipelines, ensuring data accuracy, reliability, data quality, performance, and timeliness.
• Provide expertise in Redshift database optimization, performance tuning, and query optimization.
• Assist with the design and implementation of workflows using Airflow.
• Perform data profiling and analysis to troubleshoot data-related challenges/issues and build solutions to address those concerns.
• Proactively identify opportunities to automate tasks and develop reusable frameworks.
• Work closely with the version control team to maintain a well-organized and documented repository of codes, scripts, and configurations using Git/Bitbucket.

𝐑𝐞𝐪𝐮𝐢𝐫𝐞𝐝 𝐐𝐮𝐚𝐥𝐢𝐟𝐢𝐜𝐚𝐭𝐢𝐨𝐧𝐬 𝐚𝐧𝐝 𝐒𝐤𝐢𝐥𝐥𝐬:

• Minimum 10+ years of relevant experience required.
• Advanced hands-on experience designing AWS data lake solutions, integrating Redshift with other AWS services such as DMS, Glue, Lambda, S3, Athena, and Airflow.
• Proficiency in Python programming with a focus on developing efficient Airflow DAGs and operators.
• Experience with PySpark and Glue ETL scripting, including functions like renationalize, performing joins, and transforming data frames.
• Competency in developing CloudFormation templates to deploy AWS infrastructure, including YAML-defined IAM policies and roles.
• Familiarity with debugging serverless applications using AWS tooling like CloudWatch Logs, Log Insights, and CloudTrail.
• Strong understanding of ETL best practices, data integration, data modeling, and data transformation.
• Proficiency in identifying and resolving performance bottlenecks and fine-tuning Redshift queries.
• Experience with version control systems, particularly Git, for maintaining a structured code repository.

𝐍𝐢𝐜𝐞 𝐭𝐨 𝐇𝐚𝐯𝐞 𝐒𝐤𝐢𝐥𝐥𝐬:

• Docker
• Airflow Server Administration
• Parquet file formats
• AWS Security
• Jupyter Notebooks
• API Best Practices, API Gateway, Route Structuring, and API authentication
• Git flow best practices
• Release management and DevOps
• Shell scripting
• AWS certifications related to data engineering or databases
• Experience in converting Oracle scripts and Stored Procedures to Redshift equivalents
• Proficiency in SQL programming and Redshift stored procedures for efficient data manipulation and transformation