Docker Kubernetes Spark

Python Kafka/Pulsar

AWS AWS Sagemaker Python R Scala

SQL Python S3 Hive Spark

Kafka spark Scala EC2 s3

Spark Scala Java Python SQL

Databricks Data Factory SQL Pyspark Spark

Azure Data Factory Matillion DBT Python Spark

ETL/ELT AWS Redshift Azure SQL Data Warehouse Snowflake Matillion

Kafka Java Scala Flume Python

Azure Data Factory Databricks Spark SQL

Azure Java Pyspark SQL Databricks

Azure Scala Databricks Spark ADLS/HDFS

ETL SQL AWS Spark Kafka

Scala Data Bricks DATA ARCHITECTURE Databricks Spark