Overview Apache Airflow is well renowned and widely used open-source platform for creating, scheduling, and monitoring workflows for data engineering pipelines. It offers integration with many resources whether in cloud or on-premises. Airflow workflows are represented in DAG (Direct Acyclic Graph) which manages workflow orchestration and is coded in Python scripts. …

Access your SQL database in Azure Function written in C# Overview Azure function is a serverless solution that enables enterprises to write less code, maintain less infrastructure, and hence spend less. Azure functions are event driven and can be orchestrated to solve complex problems. In this article, I will be creating…

Automate the rotation of Azure storage account access key via Azure function and update the Azure key vault secret. Overview It is always recommended to authenticate Azure resources by using managed identity. However, there are certain circumstances where this is not an option. In those scenarios, access keys, secrets, or passwords…

Run your executables or any other process using SSIS integration runtime in Azure Data Factory (ADF) When migrating SQL Server from on-premises to cloud, there is always an ask how to migrate SSIS packages or other ETL jobs to the cloud? Azure Data Factory SSIS Integration Runtime (IR) offers to…

Run your .NET executables (.exe) in Azure Data Factory using Custom activity under Batch Service Overview When migrating SQL Server from on-premises to cloud, there is always an ask how to migrate SSIS packages or other ETL jobs to the cloud? Azure Data Factory SSIS Integration Runtime (IR) offers to run…

Walk through how you can run your code inside Azure Batch and learn how to deploy application packages to compute nodes Problem Statement SQL migration from on-premises to cloud brings challenges with respect to migrating SSIS packages especially when considering PaaS services such as Azure SQL or Azure Managed Instance. Previously, I…

Migrating SSIS packages options — IaaS or PaaS — Introduction SSIS with on-premises SQL Server works differently as compared to running on Azure. Further, not all SSIS packages are written as cloud native. If we are planning to migrate on-premises SSIS packages, we need to understand our options in much detail. In broader terms, there are two major options: Migrate…

Move your local SSIS packages to Azure Data Factory Introduction SQL Server Integration Services (SSIS) emerged as an upgrade to Data Transformation Services (DTS), which was introduced with the version of SQL Server 7.0. With the release of SQL Server 2005, DTS was replaced by SSIS and has been the most…

How to setup Azure Monitor for SQL insights (Public Preview) Overview Monitoring is the very first step to optimize workload performance. It gives you a bird-eye view of the current and previous estate. Analyzing monitoring data can provide deep insights to tune the right knob for the performance. It also provides…