Accountable for data engineering lifecycle including research, proof of concepts, design, development, test, deployment and maintenance.
• Design, develop, implement, and run cross-domain, modular, optimized, flexible, scalable, secure, reliable and quality data solutions that transform data for meaningful analyses and analytics while ensuring operability
• Expert in Python and React frame works.
• Expert in Terraform and Azure Infrastructure.
• At least 8 years of active development experience with python and .NET
• At least 8 years of active development experience with UI technologies like react.
• Design, develop, implement and run data solutions and implement data pipelines with petabyte scale of data
• Layer in instrumentation in the development process so that data pipelines can be monitored. Measurements are used to detect internal problems before they result into user visible outages or data quality issues
• Build processes and diagnostics tools to troubleshoot, maintain and optimize solutions and respond to customer and production issues
• Embrace continuous learning of engineering practices to ensure industry best practices and technology adoption, including DevOps, Cloud and Agile thinking
• Skills in contributing to our industry community and strive to reuse and share components wherever possible across the organization
• Maintain high quality documentation of data definitions, transformations, and processes to ensure data governance and security
• Identifies solutions to non-standard requests and problems
• Solves moderately complex problems and/or conducts moderately complex analyses
• Ingest data from Hadoop soruces to cloud.
• Design and develop ETL workflows for migration of data from on-premises to Azure cloud
• Develop CICD wokflows on Azure for continuous deployment for projects.
• Work on databricks, Azure data factory to migrate SSIS workflows into cloud.
Bachelor's degree in Computer Science