Cloud Data Engineer

JD:

We are looking for experienced Cloud Data Engineers with strong expertise in Python and cloud data platforms to design and implement scalable data solutions.

As a Python SME, you will lead the development of cloud-based pipelines and infrastructure, collaborating with cross-functional teams to deliver reliable and efficient data systems.

Key Technical Skills:

1. Python Expertise:

Proficiency in advanced Python programming, including frameworks such as Pandas, NumPy, and PySpark.

Hands-on experience with Python for data engineering, automation, and scripting tasks.

2. Cloud Platform Skills:

AWS Cloud Data Services: Strong experience with AWS tools like S3, Lambda, Redshift, Glue, EMR, Athena, and Kinesis.

Azure Cloud Data Services: Working knowledge of Azure Data Factory, Azure CosmosDB, Azure Synapse Analytics, Azure Blob Storage, and Azure Databricks.

3. Data Engineering:

Expertise in building ETL/ELT pipelines using tools such as Apache Airflow, Prefect, or AWS Glue.

Experience in working with structured, semi-structured, and unstructured data formats like JSON, Parquet, and Avro.

Proficiency in SQL and query optimization for large-scale data sets.

4. CI/CD and DevOps:

Familiarity with CI/CD pipelines and tools such as Jenkins, GitLab, or Azure DevOps for automating deployments.

Knowledge of Infrastructure as Code (IaC) using Azure CDKTF (Terraform) or AWS CDK.

5. Data Security & Compliance:

Understanding of data governance, encryption, and compliance standards (e.g., GDPR, HIPAA).

Responsibilities:

1. Architect and implement cloud data pipelines and solutions.

2. Develop and optimize ETL/ELT workflows for data integration.

3. Collaborate with analysts and stakeholders to meet data needs.

4. Automate workflows and enhance data performance.

5. Act as an SME for Python and cloud data engineering, providing technical mentorship and guidance to team members