Data Integration Engineer

Job Description

Data Integration Engineer

Build and maintain data pipelines that are scalable, repeatable, and secure, and can serve multiple users within our company. Facilitate getting data from a variety of different sources, getting it in the right formats, assuring that it adheres to data quality, privacy and security standards, and assuring that downstream users can get that data quickly.

Description

Design and build scalable, low-latency, fault-tolerant streaming data platform that empowers diverse end users to extract meaningful and timely insights from our data assets Work closely with business and technology stakeholders to build modern distributed streaming data pipelines and analytics data stores using streaming frameworks Build a platform that facilitates gathering and collecting data, storing it, performing real-time processing on it, and serve it to end users and decision-making systems. Help drive adoption of the Data Analytics platform as part of the larger data strategy. Maintain an on-going understanding of emerging data management technologies, industry trends and best practices. Identify ways to improve data reliability, efficiency and quality. Hands-on production experience with distributed stream processing frameworks: such as Kafka / Spark Streaming / Storm Experience with deployment platforms such as Kubernetes Production experience of building a robust, fault-tolerant data pipeline that cleans, transforms, and aggregates unorganized and messy data into databases or data sources Experience with micro-services Experience with relational databases (SQL Server or Oracle, etc.) and non-relational databases Practical experience in performance tuning and optimization, bottleneck problem analysis Experience with data warehousing principles, schema design, data governance, database security is a plus Excellent communication skills Experience with agile or other rapid application development methods. Experience with object-oriented design, coding, and testing patterns as well as experience in engineering (commercial or open source) software platforms and large-scale data infrastructures. Significant knowledge of data modeling and understanding of different data structures and their benefits and limitations under particular use cases