Data Architect

Data Architect

Remote Job |   2024-02-28 17:51:13

Apply Now

Share Job 

Job Code : EWC - 1532

Team is completely Python based (Java a plus, not needed)

Need Python and Spark EMR

AWS Glue is good but comes later in migration

Needing someone Senior in Data Engineering / Architecture

Come in and drive solution design at team level and beyond

Nobody interviewed was high level enough to feel they could do this

• This year, data tech team is going through migration to converge current platforms into Spark and Glue ETL
• Expertise in Batch ETL
• Apache Spark is current work, Glue comes later based on priority
• Converging a lot of existing jobs, Spark and non-Spark jobs, Apache Spark running on EMR, eventually onto Glue. Coming in with Glue is helpful ,but main expertise in Spark EMR is needed first. They want to get to the point where everything is running on Glue, but that's a little ways away. First is getting Spark EMR jobs in place.