Senior Data Engineer

Roles And Responsibilities

Owning the design, operations and improvements for the Customer Data Warehouse infrastructure
Engineering solutions to aggregate and automate large scale data flows from varying sources
Collaborate with others to construct complex data sources for algorithms and machine learning models
Gather business requirements from business stakeholders and deliver a solution.
Participate in weekly status meetings to provide updates to stakeholders and provide a strategic plan for upcoming tasks.
Explore and learn the latest Azure technologies to provide new capabilities and increase efficiency
Build real-time streaming from internal and external sources to provide insights into the business
Help continually improve ongoing reporting and analysis processes, automating or simplifying self-service support for customers.
Create and maintain workflow/technical documents for all the development activities

Specific Skills Required Are

Experience in implementing data solutions in Azure including Azure SQL, Azure Synapse, Cosmos DB, Databricks.
Hands-on experience configuring Delta Lake on Azure Databricks.
Experience in designing data solutions in Azure including data distributions and partitions, scalability, disaster recovery and high availability.
Hands on Development or Architecture experience using Azure SQL or SQL DWH (synapse)
Knowledge of batch and streaming data architectures
Experience using big data technologies (Hadoop, Hive, HBase, Spark and others).
Strong knowledge of SQL (MSSQL or MySQL or PostgreSQL or Presto).
Demonstrated strength in data modeling, ETL development, and data warehousing
Hands-on experience in Data Engineering using Python.
Good knowledge of data lake optimization, partitioning, cron job scheduling and debugging.
Good knowledge of file formats like parquet, CSV and Hadoop file systems.

General Requirements

Bachelors or Master's Degree from a reputed college
5+ Years of experience in the Data Engineering domain; Worked in the controlled yet agile setup using proven project management methodologies
Experience in all stages of DE project work (requirements and logical design, physical design,

implementation, testing and deployment)

Experience communicating with senior management as well as with colleagues from engineering, analytics, and business backgrounds
Capable of performing under minimal supervision, multi-tasking, meeting deadlines as an individual contributor, and coaching a team.