Lead Databricks Engineer

Candidates are required to have these mandatory skills:

•            Strong hands on in Pyspark and Apache Spark.

•            Experience in Native Spark Migration to Databricks.

•            Experience in Hadoop Migration to Databricks.

•            Experience in Building Data Governance Solutions like Unity Catalog, StarBust …etc

•            Build Very Strong Orchestration Layer in Databricks/ADF…. Workflows.

•            Build CICD for Databricks in Azure Devops.

•            Process near Real time Data thru Auto Loader, DLT Pipelines.

•            Implement Security Layer in Delta Lake.

•            Implement Massive Parallel Processing Layers in Spark SQL and PySpark.

•            Implement Cost effective Infrastructure in Databricks.

•            Experience In extracting logic and from on prem layers like SSIS, Stored procedures, Informatica, Vertica, Apache Hudi, Filesystems. etc into Pyspark.

•            Ability to Build Solutions as Cloud Agnostic.

•            Proven experience in designing and implementing complex data solutions aligned with business objectives.

•            Expertise in data modelling, integration, security, and governance

•            Hands-on experience with guiding the virtual data model definition, defining Data Virtualization architecture and deployment with focus on Azure, Databricks, PySpark technologies.

•            Prior experience with establishing best practices for business optimizations.

•            Experience with relational and non-relational data stores (Hadoop, SQL, Mongo DB), ETL or ELT tools (SSIS, Informatica, Matillion, DBT), DevOps, Data Lake and Data Fabric concepts

•            In-depth experience with data governance, data integration and related technologies.

•            Proficiency in a variety of database technologies, both relational and non-relational.

•            Knowledge of cloud-based data solutions (e.g., AWS, Azure).

•            Excellent collaboration and communication skills