Sr. Data Engineer

Job Description

• Teradata Expertise:
  • In-depth knowledge of Teradata architecture and components (e.g., AMP, Nodes, Parsing Engines, etc.).
  • Proficiency in Teradata SQL and Teradata utilities.
• SQL and Scripting:
  • Strong SQL skills for querying and managing relational databases.
  • Scripting experience (e.g., Shell, Python) for automating tasks.
  • Know tools like Teradata Utilities (e.g., BTEQ, FastLoad, MultiLoad, TPT, etc.).
• ETL Tools:
  • Experience with ETL tools and technologies used in Teradata environments.
• Performance Optimization:
  • Knowledge of query optimization techniques, indexing, partitioning, and workload management in Teradata.
• Data Warehousing Concepts:
  • Strong understanding of data warehousing principles, including star/snowflake schemas, OLAP, and ETL processes.
• Cloud Platform / AWS Expertise:
  • Proficiency with core AWS services: EC2, S3, RDS, Lambda, CloudWatch, CloudFormation, etc.
  • Experience with AWS networking (VPC, subnets, security groups, NACLs) and security management.
  • Familiarity with AWS storage options, such as S3, EFS, and Glacier, in the context of large-scale data storage and retrieval.
• Teradata and AWS Integration:
  • Experience deploying and managing Teradata in AWS cloud environments.
  • Knowledge of AWS services that enhance Teradata performance, such as using S3 for staging data or integrating with Redshift for analytics.
• Automation and Infrastructure as Code (IaC):
  • Hands-on experience with AWS CloudFormation, Terraform, or similar tools for automating infrastructure deployment.
  • Proficiency in automating ETL pipelines and data processing workflows using AWS Lambda, Step Functions, or AWS Glue.
• Data Security in AWS:
  • Understanding of encryption and security protocols for data stored and processed in AWS.
  • Experience implementing compliance standards such as GDPR, HIPAA, or other data protection regulations in cloud environments.

Responsibilities:

• Database Management:
  • Design, implement, and maintain Teradata databases.
  • Optimize and tune queries for performance.
  • Monitor and troubleshoot database issues.
• ETL Development:
  • Develop and maintain ETL (Extract, Transform, Load) processes for data integration and transformation.
  • Use tools like Teradata Utilities (e.g., BTEQ, FastLoad, MultiLoad, TPT, etc.).
• Data Modeling:
  • Create and maintain data models to support business processes.
  • Work with stakeholders to understand data requirements and design solutions.
• Performance Tuning:
  • Identify and resolve performance bottlenecks in SQL queries, system configuration, or data design.
  • Regularly monitor and optimize system performance.
• Data Security & Governance:
  • Implement data security best practices.
  • Ensure compliance with data governance policies and regulations.
• Cloud Integration and Management:
  • Implement and manage Teradata on AWS infrastructure, leveraging AWS services for scalability and performance.
  • Manage data migration from on-premises Teradata environments to AWS cloud-based Teradata instances.
  • Ensure seamless integration between Teradata and AWS services like S3, Redshift, RDS, and Lambda.
• Cloud Infrastructure:
  • Design and deploy cloud infrastructure (VPCs, EC2 instances, S3 buckets) to support Teradata environments.
  • Implement networking and security best practices in AWS to ensure secure data access and storage.
• Automation and DevOps:
  • Automate infrastructure provisioning, scaling, and monitoring using AWS tools like CloudFormation, Terraform, or AWS CLI.
  • Use AWS Lambda for automating ETL processes or handling specific workloads.
• Data Warehousing and Analytics:
  • Leverage AWS data services (e.g., Redshift, Athena, Glue) to complement Teradata workloads.
  • Implement data pipelines using AWS services for ETL processes, integrating data from various sources into Teradata.
• Security and Compliance:
  • Implement AWS security best practices, such as IAM roles and policies, to secure Teradata environments in the cloud.
  • Ensure compliance with relevant data governance policies, including encryption at rest and in transit.
• Collaboration:
  • Work closely with data architects, analysts, and other stakeholders to understand data needs and deliver solutions.