Senior Big Data Engineer

Job Title: Senior Big Data Engineer
Location: Phoenix, AZ
Client: American Express (AMEX)
Responsibilities:
• Design, develop, and maintain scalable and high-performance data processing pipelines using big data technologies such as Hadoop, Spark, Kafka, and Flink.
• Architect and implement data storage solutions that can handle large volumes of structured and unstructured data effectively.
• Collaborate with data scientists and analysts to understand business requirements and translate them into technical solutions.
• Optimize and tune data processing workflows for performance and efficiency.
• Implement data quality checks and validation processes to ensure data accuracy and reliability.
• Work closely with cross-functional teams to integrate data from various sources and systems.
• Stay updated with the latest trends and advancements in big data technologies and propose innovative solutions to enhance our data infrastructure.
• Provide technical leadership and mentorship to junior members of the team.
• Participate in code reviews, architectural discussions, and brainstorming sessions to drive continuous improvement.
Requirements:
• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• Minimum of 10 years of professional experience in big data engineering.
• Proficiency in programming languages such as Java, Scala, or Python.
• Extensive experience with distributed computing frameworks like Hadoop MapReduce, Apache Spark, Apache Flink, etc.
• Strong knowledge of big data storage technologies such as HDFS, Apache HBase, Amazon S3, etc.
• Hands-on experience with stream processing frameworks like Apache Kafka.
• Solid understanding of cloud platforms such as AWS, Azure, or GCP.
• Experience with containerization technologies like Docker and orchestration tools like Kubernetes.
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration skills.
• Ability to work effectively in a fast-paced and dynamic environment.
• Certifications in big data technologies (e.g., Cloudera, Hortonworks, Databricks) are a plus.