Senior Data Engineer

Job Description

Health Care Company, Inc. (Health Care Company) is looking for a Senior Data Engineer to work 100% remote (will work 9am-5pm either AZ time zone or EST time zone) to design, develop, test, and maintain pipelines ingesting critical real-time claims data sets into OnePaaS (Cloud)

Accountable for data engineering lifecycle including research, proof of concepts, design, development, test, deployment, and maintenance

Design, develop, implement and run cross-domain, modular, optimized, flexible, scalable, secure, reliable and quality data solutions that transform data for meaningful analyses and analytics while ensuring operability

Design, develop, implement and run data solutions and implement data pipelines with petabyte scale of data.

Build processes and diagnostics tools to troubleshoot, maintain and optimize solutions and respond to customer and production issues.

Embrace continuous learning of engineering practices to ensure industry best practices and technology adoption, including DevOps, Cloud and Agile thinking

Maintain high quality documentation of data definitions, transformations, and processes to ensure data governance and security.

Identifies solutions to non-standard requests and problems.

Solves moderately complex problems and/or conducts moderately complex analyses.

Ingest data from Hadoop sources to cloud.

Design and develop ETL workflows for migration of data from on-premise to Azure cloud

Develop CI/CD workflows on Azure for continuous deployment for projects

Work on Databricks, Azure data factory to migrate SSIS workflows into cloud.

Requirements

5 years of experience on Azure cloud

Python or Scala experience

Azure Databricks

Azure Data Factory (ADF)

UI or API build experience (React or Angular)

Kafka for real-time integration

Bachelor's Degree

Desired Skills and Experience

AZURE DATABRICKS, AZURE DATA FACTORY, ADF, AZURE CLOUD, REACT, ANGULAR, KAFKA, PYTHON, SCALA