Sr Data Engineer

Responsibilities

Design, Develop, and Maintain Data Pipelines:

Utilize Databricks and Spark, along with other cloud technologies as needed.

Optimize data pipelines for performance, scalability, and reliability.

Ensure Data Quality And Integrity

Maintain data quality and integrity throughout the data lifecycle.

Collaborate With Stakeholders

Work closely with data scientists, analysts, and other stakeholders to understand and meet their data needs.

Troubleshoot And Resolve Data Issues

Identify and resolve data-related issues.

Provide root cause analysis and recommendations.

Documentation And Communication

Document data pipeline specifications, requirements, and enhancements.

Communicate effectively with the team and management.

Data Validation And Analysis Tools

Create new data validation methods and data analysis tools.

Share best practices and learnings with the data engineering community.

Implement ETL Processes And Ensure Compliance

Implement ETL processes and data warehouse solutions.

Ensure compliance with data governance and security policies.

Qualifications

Bachelor's degree in computer science, Engineering, or related field, or equivalent work experience.

3+ years of experience in data engineering, preferably with Databricks and Spark.

Proficient in SQL and Python, familiar with Java or Scala.

Experience with cloud platforms, such as Azure or AWS.

Experience with big data technologies, such as Kafka, Hadoop, Hive, etc.

Familiarity with data warehouse and data lake concepts and architectures.

Experience with data integration and ETL tools, such as Azure Data Factory or Talend.

Familiarity with data visualization and reporting tools, such as Power BI or Tableau.

Strong analytical and problem-solving skills.

Excellent communication and teamwork skills.
 

Desired Skills and Experience

SENIOR DATA ENGINEER, SENIOR BIG DATA ENGINEER, SR. BIG DATA ENGINEER, BIG DATA ENGINEER, DATA ENGINEER, DATA PIPELINE, DATA PIPELINES, PIPELINES, PIPELINES, BIGDATA, BIG DATA

• , KAFKA, HADOOP, HIVE, CLOUD, AWS, AZURE, SQL, PYTHON, JAVA, SCALA