Course Content
Data Engineering Services on Azure
- Azure Datalake Storage (ADLS) gen2
- Azure Blob Storage
- Azure SQL database
- Azure Data Factory
- Azure Databricks
- Azure Synapse Analytics
- Microsoft Fabric
Data Storages on Azure
- File stores
- Relational data store
- Non-Relational data stores
- Azure Blob storage
- NoSQL & Azure Cosmos DB
- Azure Data Lake Storage
- Why Data Lake
- Data Lake architecture
- Azure SQL Database
- Azure database for MySQL
- Azure database for MariaDB
- Azure database for PostgreSQL
- Azure Synapse Analytics
Azure Data Factory (ADF)
- Introduction to Azure Data Factory (ADF)
- ETL vs ELT
- On-prem vs Cloud ETL tools
- Why ADF
- ADF Components
- Linked Services in ADF
- Integration Runtime (IR): Self-hosted / Azure
- Working with various Copy data scenarios
- Working with Azure SQL database
- Working with Control flow activities
- Working with Data Flow Transformations
- Working with Parameters in ADF
- SCD Type 1 & 2 Pipelines
- ADF Data Migration
- SSIS
- Pipelines and activities: Copy data, Control flow, Data Flow
- Scheduling & Triggering Pipelines
- Monitoring Pipelines
- CI/CD in ADF
- ADF Projects
- Data Ingestion methods: Full load, Copy data tool, Batch load, Sequential load, Incremental load, CDC
Python
- Variables
- Datatypes in Python
- Operators in Python (Arithmetic, Logical, Comparison)
- Working with Strings
- Indexing & Slicing
- Data Structures in Python (List, Tuples, Sets, Dictionaries)
- Functions (Built-in, User defined)
- Conditional Flow Statements (If condition, While loop, For iteration, Range() function)
- Map, Filter, Reduce, Lambda expressions
Apache Spark
- Introduction to Apache Spark
- Hadoop vs Spark
- Understanding Spark Architecture
- Spark Context
- Spark APIs
- Spark Data structures (RDDs, DataFrames, Datasets)
- Creating Spark RDDs & DataFrames
- Column manipulations
- Partitions & Parallelism
- Reading & Writing data
- PySpark commands
- Spark SQL
- Group by & Aggregate Functions
- Spark Transformations & Actions
- Lazy Evaluation & Fault Tolerance
- Window Functions
- Joins & Strategies
- Conditional Functions
- Caching
- Schema evolution
- Data Skew
- Catalyst Optimizer
Data Engineering with Azure Databricks
- Introduction to Azure Databricks
- Azure Databricks vs Databricks Community Edition
- Databricks Runtime Engine
- Databricks Components (DBFS, Workspace, Cluster, Catalog, Tables, Hive Metastore)
- Working with Files using DBUTILS
- Creating Tables (using UI, using Notebooks)
- Reading data from different files & Writing data to different formats
- Visualization in Databricks Notebooks
- Integrating Azure Blob & ADLS with Databricks
- Mounting with Access Keys, Secret Scopes using Azure Key Vault, Service Principle, SAS Tokens
- Delta Lake & Creating Delta Tables (Managed tables, External tables)
- Lakehouse & Medallion Architecture (Bronze layer, Silver layer, Gold layer)
- Scheduling Databricks Jobs using ADF Pipelines
- Delta Live Tables (DLT)
- Unity Catalog
Snowflake Fundamentals
- Introduction to Snowflake & Cloud Data Warehousing
- Snowflake Setup & Architecture
- Databases, Schemas, Tables & Views
- Internal & External Stages
- COPY command for Data Loading
- Snowpipe & Continuous Ingestion
- Zero-Copy Cloning
- Snowflake vs Databricks & Snowflake vs Traditional DWH
- Virtual Warehouses
- Roles, Users & Access Control
- File Formats & Patterns
- Data Integration & Connectivity
- Time Travel & Versioning
- Security & Governance
Apache Airflow
- Introduction to Apache Airflow
- Airflow & Docker Installation
- Directed Acyclic Graphs (DAGS)
- Tasks, Operators, Hooks & Providers
- Task Groups & Dependency Management
Hands-on Real-time DE Scenarios
- Creating ETL/ELT Pipelines in Azure Data Factory
- Working with Control Flow activities in ADF Pipelines
- Working with Data Flow Transformations in ADF Pipelines
- Ingesting multiple File sources using ADF pipelines
- Ingesting multiple SQL tables using ADF Pipeline
- Implementing Batch load & Sequential load
- Incremental loading using last modify date & time
- Incremental loading and Change data capture
- SCD1 pipeline in ADF
- SCD2 pipeline in ADF
- Analyzing data with PySpark & Spark SQL
- Analyzing ADLS data from Databricks Notebooks
- Mounting & Service Principle
- Creating Delta Tables
- Lakehouse (Medallion) architecture
- Building Pipelines using AWS
- Snowflake scenarios
Real-Time Projects
- ADF Data Migration Project (migrating on-prem SQL Server data to Azure)
- Incremental loading using Last Modified Date/Time
- SCD-2 Pipeline in ADF
- E-commerce Data Pipeline using Spark & Databricks
- Finance Project using Docker, Airflow, Snowflake & AWS
- Real-Time Streaming Analytics using Azure
Bonuses & Support
- Interview Questions and Answers
- Mock Interviews
- Placement Assistance
- Certification Support
- Resume/CV Preparation
- LinkedIn Profile Optimization