Big Data Engineer

Job Description:
We are seeking a talented Big Data Engineer to join our technology team at American Express. As a Big Data Engineer, you will be responsible for designing, building, and maintaining scalable data pipelines and infrastructure to support our big data initiatives. You will work closely with data scientists, analysts, and other stakeholders to ensure our data platforms are robust, efficient, and capable of handling large volumes of data.
Design, develop, and implement end-to-end data pipelines using big data technologies such as Hadoop, Spark, Kafka, etc.
Build and maintain scalable data ingestion and transformation processes to support data lake and data warehouse solutions.
Collaborate with data scientists and analysts to understand data requirements and implement solutions that meet business needs.
Optimize and tune data pipelines for performance, reliability, and scalability.
Develop and implement data governance best practices and standards.
Implement data security and privacy measures to protect sensitive information.
Monitor and troubleshoot data pipeline issues and provide timely resolutions.
Stay current with emerging technologies and industry trends related to big data and analytics.
Requirements:
Bachelor's degree in Computer Science, Engineering, or a related field. Master's degree is a plus.
Proven experience as a Big Data Engineer or similar role, with hands-on experience in designing and implementing large-scale data solutions.
Strong programming skills in languages such as Python, Scala, or Java.
Experience with big data technologies and frameworks (e.g., Hadoop, Spark, Kafka, HBase, Hive, etc.).
Proficiency in SQL and NoSQL databases.
Experience with cloud platforms such as AWS, Azure, or Google Cloud Platform.
Familiarity with data warehousing concepts and ETL processes.
Understanding of data governance, data security, and data quality best practices.
Excellent problem-solving and analytical skills.
Strong communication and collaboration skills.
Preferred Qualifications:
Certification in big data technologies or cloud platforms.
Experience with containerization and orchestration tools (e.g., Docker, Kubernetes).
Knowledge of machine learning and data science concepts.
Financial services industry experience is a plus.