Data engineer

Job description:

• Ability to troubleshoot and diagnose complex issues
• Able to demonstrate experience supporting technical users and conduct requirements analysis
• Can work independently with minimal guidance & oversight
• Experience with IT Service Management and familiarity with Incident & Problem management
• Highly skilled in identifying performance bottlenecks, identifying anomalous system behavior, and resolving root cause of service issues.
• Demonstrated ability to effectively work across teams and functions to influence design, operations, and deployment of highly available software
• Knowledge of standard methodologies related to security, performance, and disaster recovery

Required Technical Expertise
• Expertise in language such as Java, Python. Implementation knowledge in data processing pipelines using programming languages like Java and Python to extract, transform, and load (ETL) data
•             Create and maintain data models, ensuring efficient storage, retrieval, and analysis of large datasets
•             Troubleshoot and resolve issues related to data processing, storage, and retrieval.
•             3-5 years’ Experience in designing, developing, and deploying data lakes using AWS native services (S3, Glue (Crawlers, ETL, Catalog), IAM, Terraform, Athena)
•             Experience in development of systems for data extraction, ingestion and processing of large volumes of data
• Experience with data pipeline orchestration platforms
• Experience in Ansible/Terraform/Cloud Formation scripts and Infrastructure as Code scripting is required
• Implement version control and CI/CD practices for data engineering workflows to ensure reliable and efficient deployments
•Proficiency in implementing monitoring, logging, and alerting solutions for data infrastructure (e.g., Prometheus, Grafana)
• Proficiency in distributed Linux environments

Preferred Technical Experience
• Familiarity with data streaming technologies such as Kafka, Kinesis, spark streaming, etc
• Knowledge of cloud platforms (prefer AWS) and container + orchestration technologies
• Experience with AWS OpenSearch, Splunk
• Experience with common scripting and query languages.