Data engineer (Azure)

Job Description:
We are seeking a Senior Data Engineer to join our team and contribute to the E2E (End to End) Data Quality Framework initiative within the FVD (Fix Vehicle Data) domain. The successful candidate will play a crucial role in addressing pain points related to data capture, data quality (completeness, accuracy, and consistency), and data lineage within the OnStar Transformation Fix Vehicle Data initiative. The candidate will work closely with cross-functional teams to ensure the delivery of high-quality vehicle data from the back office to various stages such as Vehicle Communication Services, Vehicle Data Factory, Vehicle External Data Consumption, Telematics 360, and outbound to Azure and customers.

What You'll Do:
•   Focus on addressing data capture issues before data collection and delivery.
• Provide insights into potential data capture issues and facilitate proactive resolution.
• Implement DQ rule engine solutions using Spark/Scala in Azure for real-time streaming and micro-batch processing.
• Evaluate and assess the suitability of the Spark/Scala solution, considering factors such as scalability and ease of implementation.
• Collaborate with the business partners to analyze their existing solution and enhance it to meet specific needs.
•  Build and maintain monitoring dashboards for tracking latency and ensuring efficient data transformation.
• Develop a tool to trace data lineage throughout the data pipeline.
• Enable visibility into source-to-target mapping throughout the data pipeline. .
•  Collaborate with cross-functional teams to ensure effective monitoring and notification mechanisms.
Your Skills & Abilities (Required Qualifications):
•   Bachelor's or Master's degree in Computer Science, Software Engineering, or a related field.
• Strong experience as a Senior Developer with expertise in data quality frameworks and pipeline architectures.
•  Proficiency in programming languages such as Scala, Python, Java, and SQL.
•   Hands-on experience with Spark and Scala for real-time streaming and micro-batch processing.
•  Knowledge of data governance, data cataloging, and data lineage concepts.
•  Experience with Azure (preferred) or AWS cloud platform for data processing and storage.
•Excellent problem-solving skills and the ability to work in a fast-paced, collaborative environment.
•  Strong communication skills to effectively collaborate with cross-functional teams and stakeholders.
What Will Give You a Competitive Edge (Preferred Qualifications but any 1 is required):