Hadoop Developer

Responsibilities:

• Responsible for developing distributed computing Big Data applications using Hadoop, Hive, SPARK, SPARKSQL, Sqoop, Kafka, Spark Streaming and Zookeeper.
• Works with Hive tables by implementing partitioning and bucketing, writing and optimizing HiveQL queries for high performance and throughput.
• Handling different file formats like PARQUET, ORC, Avro, CSV, JSON file formats.
• Analyzing text, streams with emerging Hadoop-based Big Data, NoSQLs such as HBASE.
• Responsible for designing, developing, testing, tuning and building a large-scale data processing system, for Data Ingestion and Data products that allow the Business application to improve quality, velocity and monetization of data assets for both Operational Applications and Analytical needs.
• Creating data platform that supports the Claim team management in performance reporting and managing daily operations by providing timely and accurate information aimed to reduce administrative costs and improve quality of claims for members.
• Processing Claims/Care data by applying transforms using Spark to Sailfish.
• Developing ETL application for extracting the data from DB2 Sources and pushing the files through Secure Transfer to various third-party vendors.
• Works with Business Analysts to transform use cases into Hadoop/Bigdata centric implementation
• Working on Hortonworks Distribution and Amazon Web Services S3 for Hadoop implementation.
• Uses GIT HUB for centralized code repository, code review and version control tool.
• Develop Spark jobs in Scala for data processing and data analysis
• Configures Control-M for scheduling Batch and streaming jobs.
• Develops Spark Streaming jobs with Kafka for real time data ingestion.
• Create and maintain HBase tables to store time series data and manage data compaction for efficient performance.
• Create change record with all the tasks needed for production deployment.
• Perform demos, design reviews, and code reviews for program and platform-level governance groups.
• Participates in sprint planning sessions, inc luding providing detailed tasks that are required to fulfill user stories.

Degree Requirement:

Bachelor’s degree in computer science, computer information systems, information technology, or a closely realted IT field, or a combination of education and experience equating to the U.S. equivalent of a bachelor’s degree in one of the aforementioned subjects