Databricks Administrator

Responsibilities

• Configure, deploy and maintain Databricks clusters and workspaces.
• Use Terraform to provision and deploy Databricks clusters, workspaces, and associated AWS resources. Ensure consistency and repeatability by treating infrastructure as code
• Monitor cluster performance, resource utilization, and troubleshoot issues to ensure optimal performance.
• Implement and manage access controls and security policies to protect sensitive data.
• Manage platform feature and software releases and updates.
• Provide technical support to Databricks users, including troubleshooting and issue resolution.
• Conduct training sessions to educate users on platform best practices and capabilities.
• Optimize cluster configurations for performance and cost-effectiveness.
• Monitor and manage auto-scaling to ensure resources are allocated efficiently.
• Implement and enforce security policies, RBAC, access controls, and encryption mechanisms.
• Develop and maintain backup and disaster recovery strategies to ensure data integrity and availability.
• Collaborate with data engineers to integrate Databricks with other data sources, data warehouses, and data lakes.
• Monitor and manage platform costs, including optimizing resource utilization to control expenses.
• Collaborate with cross-functional teams, including data scientists, data engineers, and business analysts to understand their requirements and provide technical solutions