BigData Platform Engineer/Architect

Summary:

The Platform Engineer /Architect is responsible to define architecture and strategy for Batch workloads in Hybrid cloud environment. This includes POC design, enabling multitenancy and capacity planning at enterprise scale, component stress and load testing, deployment automation, monitoring and governance.
 
Responsibilities:

The main and routine tasks of this position are to:
 

• Design and develop the Batch data processing on onprem or public cloud
• Develop and maintain an advanced platform to manage Kubernetes clusters lifecycle for batch processing.
• Understand the On prem VM based Bigdata ecosystem, designing and developing migration plan to on prem cloud native deployment.
• Determine security and segregation of Lab and Factory/BAU workloads.
• Develop tools for automated build, test, deployment and management of the platform.
• Determine approach for multitenancy and capacity planning at enterprise scale.
• Improve continuous integration and delivery systems.
• Monitor system events to ensure health, maximum system availability and service quality
• Maintain documentation regarding configurations, operations and troubleshooting procedures
• Participate in the definition of standards, guidelines and best practices.
• Assist in evaluating new requirements, technical design and standards.
• Participate in R&D effort bringing new ideas and new technologies

 
Qualifications

A University Degree or equivalent experience in Computer Information Systems, Computer Science, Mathematics or a related field.
 
Relevant experience:
 

• Architect: Have worked on Big Data for more than 8+ years and have hands on with Spark, Hive & Presto.
• Developer: 2+ years of experience in Kubernetes/OpenShift.

 
Skills:
 

• Good with Architecture and has good understanding and in depth knowledge on distributed systems 
• Professional experience building enterprise Big Data Applications using Spark, Hadoop, Hive etc.                
• Hands on Knowledge on Containerisation and deploying Opensource products on K8s/ OpenShift.
• Maintaining and building inhouse based Opensource platforms like Hadoop, Spark, Hive, Presto
• Skill in developing techniques and methodologies to resolve unprecedented problems or situations
• Ability to communicate effectively with all levels of management
• Ability to communicate with remote teams
• Ability to make complex information accessible
• Must be a self-starter that requires only limited supervision/guidance
• Candidate must present with honesty, integrity and maintains confidentiality at all times
• Be a team player
• Have good interpersonal communication skills
• Ability to work effectively in a fast paced changing environment
• Ability to multi-task and prioritize
• Thorough experience in defining and applying design standards, patterns, quality matrix depending on the solutions
• Deep knowledge of OOPs, data structure, and algorithm
• Ability & willingness to learn technologies at pace

 
Knowledge:
 

• Expertise with Java or any coding languages.
• Linux Scripting (Bash, Python, etc.)
• Expertise with Source Control Management (Git, Bitbucket, etc.)
• Expertise with CI/CD (Jenkins/ArgoCD)
• Expertise with Configuration Management (Ansible, Terraform etc.)
• Expertise with Monitoring (Prometheus, etc.)
• In-depth knowledge with Linux environments (RHEL)
• Knowledge on Protegrity.
• Knowledge of Cloud infrastructures (AWS EKS, Google GKS, Azure AKS, etc.)
• Knowledge of Web Servers (Nginx, Apache, etc.)
• Knowledge in RESTful API design and implementation
• Knowledge in the development of high-performance and fault-tolerant systems
• Knowledge in distributed systems programming
• Experience with Java or Go programming language an asset
• Experience with Docker an asset
• Experience running Kubernetes in production an asset
• Experience in large-scale production deployments an asset