Lead Databricks Engineer

Job Description:       

• Strong hands on in Pyspark and Apache Spark.
• Experience in Native Spark Migration to Databricks.
• Experience in Hadoop Migration to Databricks.
• Experience in Building Data Governance Solutions like Unity Catalog, StarBust …etc
• Build Very Strong Orchestration Layer in Databricks/ADF…. Workflows.
• Build CICD for Databricks in Azure Devops.
• Process near Real time Data thru Auto Loader, DLT Pipelines.
• Implement Security Layer in Delta Lake.
• Implement Massive Parallel Processing Layers in Spark SQL and PySpark.
• Implement Cost effective Infrastructure in Databricks.
• Experience In extracting logic and from on prem layers like SSIS, Stored procedures, Informatica, Vertica, Apache Hudi, Filesystems. etc into Pyspark.
• Ability to Build Solutions as Cloud Agnostic.
• Proven experience in designing and implementing complex data solutions aligned with business objectives.
• Expertise in data modelling, integration, security, and governance
• Hands-on experience with guiding the virtual data model definition, defining Data Virtualization architecture and deployment with focus on Azure, Databricks, PySpark technologies.
• Prior experience with establishing best practices for business optimizations.
• Experience with relational and non-relational data stores (Hadoop, SQL, Mongo DB), ETL or ELT tools (SSIS, Informatica, Matillion, DBT), DevOps, Data Lake and Data Fabric concepts
• In-depth experience with data governance, data integration and related technologies.
• Proficiency in a variety of database technologies, both relational and non-relational.
• Knowledge of cloud-based data solutions (e.g., AWS, Azure).
• Excellent collaboration and communication skills