Data Engineer

Job Description;
Proficient IT professional experience with 8+ years of expertise as a Data Engineer, ETL Developer and Software Engineer includes designing,
developing, & implementing data models for enterprise-level applications. 
Worked on ETL Migration services by creating & deploying AWS Lambda functions to provide a serverless data pipeline that can be written

to Glue Catalog & queried from Athena. 
Experience in Analytics & cloud migration from on-premises to AWS Cloud with AWS EMR, S3, & DynamoDB.
Experience in creating & managing reporting & analytics infrastructure for internal business clients using AWS services including Athena,

Redshift, Spectrum, EMR, & Quick Sight. 
Extensive expertise with Amazon Web Services such as Amazon EC2, S3, RDS, IAM, Auto Scaling, CloudWatch, SNS, Athena, Glue, Kinesis,

Lambda, EMR, Redshift, & DynamoDB. 
Proficiency in setting up the CI/CD pipelines using Jenkins, GitHub, Chef, Terraform & AWS.
Created an Azure SQL database, monitored it, & restored it. Migrated Microsoft SQL server to Azure SQL database.
Experience with Azure Cloud, Azure Data Factory, Azure Data Lake Storage, Azure Synapse Analytics, Azure Analytical services, Big Data

Technologies (Apache Spark), & Data Bricks. 
Developed ETL pipelines in & out of the data warehouse using a mix of Python & Snowflakes, SnowSQL Writing SQL queries against Snowflake.
Extensive experience developing & implementing cloud architecture on Microsoft Azure.
Excellent understanding of connecting Azure Data Factory V2 with a range of data sources & processing the data utilizing pipelines, pipeline

parameters, activities, activity parameters, & manually/window-based/event-based task scheduling. 
Created a connection from Azure to an on-premises data center using the Azure Express Route for Single & Multi-Subscription.
Working knowledge in Python programming with a variety of packages such as NumPy, Matplotlib, SciPy, & Pandas.
Integrated Jenkins with Docker container using Cloud bees Docker pipeline plugin & provisioned the EC2 instance using Amazon EC2 plugin
Extensive experience creating Web Services with the Python programming language, including implementation of JSON-based RESTful & XML-

based SOAP web services. 
Experienced in writing complex Python scripts with Object-Oriented principles such as class creation, constructors, overloading, & modules.
Experience establishing & maintaining multi-node development & production Hadoop clusters. 
Worked with Spark to improve the speed & optimization of current Hadoop algorithms utilizing SparkContext, Spark-SQL, Data Frame, Pair RDD,

& Spark YARN.

Worked with the Map Reduce programming paradigm & the Hadoop Distributed File System.


 

Desired Skills and Experience
Experience with Azure Cloud, Azure Data Factory, Azure Data Lake Storage, Azure Synapse Analytics,Azure Analytical services, Big Data Technologies (Apache Spark), & Data Bricks.