Azure Data Factory V2, Azure Data Lake, Azure Databricks, SQL Synapse Analytics, Azure Storage - Blob + ADLS, Azure SQL
What you'll learn
- Insights of Azure Data Services - Storage Services, ADF V2, Azure Databricks and SQL DW (Synapse Analytics)
- Creating and Managing data related Azure services
- Practical experience with downloadable datasets
- Design concepts for big data processing on Azure Data Lake
- Securing Data on Blob Storage using SAS URLs + Lifecycle Policies to archive data with auto tier change
- End to End Data load pipeline (Source -> Raw Layer -> Curated -> Consumption)
Requirements
- ETL or Database Knowledge is expected
- Basic Azure Knowledge is plus
- Spark knowledge will speed up learning Azure Databricks
Description
Learn All in One Azure Data Engineering services along with hands-on (requirement gathering to implementation)
------------------------------------------------------------------------------------------------------------------
Recently added sessions:
- Added code snippets for Databricks <-> SQL Server using JDBC
- Data Generator code added. Now you can generate your own datasets for this workshop.
- Copy data from Azure SQL to Azure SQL without ADF (kind of linked server)
- ADF Code promotion - manual activity
- Dynamically change DB connection in ADF (Parameterizing linked service)
- Visible comparison of Blob and Azure Data Lake Gen2 storage using data analytics through Azure Databricks.
- NY Taxi data files source link added in resources section
- Integrate Key Vault with Azure Databricks and extract secrets via Secret Scope
- Use Blob SDK in Azure Databricks to read/write files on blob like IO operations
- Connecting to SQL Server with pyodbc in Azure Databricks
- Datasets used in this course are now available to download. Visit Azure Data Factory section.
We will execute a small project. It will help you to learn:
1) Design concepts - How to design big data system. Terminologies of each components/layers.
2) Hands-on - We will be creating required Azure data services and will look various options available.
3) Tips and Tricks
4) Helps in DP-200 and DP-201 certification preparation.
Technologies you will learn:
Azure Data Factory V2, Databricks (using Standard cluster), Storage Services, SQL Synapse Analytics (DB part)
Course highlights:
1) Resources: Links to external resource for further reading
2) Datasets : Download and use dataset used in this course. Do more practice.
3) Architecture and Design : Start learning on how to deign the data load on cloud platform.
Who this course is for:
- Beginners to Azure Data World
- ETL Developers
- Database Professionals
