Retries are a useful part of resilient pipelines. However, retries aren’t created equal, so I want to dive into recent changes to pipeline activity retries and the benefits they bring. The issue Failures happen, so we have retries to have another chance (or few) at completing the activity. They look something like this: These fixed […]
Pipeline sequences are a common orchestration pattern, and chaining them allows us to decompose processes into manageable chunks. Here I want to look at 3 ways to achieve this chaining in Fabric and how they compare. Invoke pipeline The default option is the native Invoke Pipeline activity. This allows us to call pipelines wherever they […]
Whether running Data Factory, Synapse, or Fabric pipelines, things go wrong – and the de facto response is to send an email. We’ve looked at sending emails from pipelines before, but at scale they can become noise and are easy to ignore. A more effective option is to surface alerts where collaboration already exists, such as Teams. […]
If you have a Self-Hosted Integration Runtime (SHIR, or IR for short here) on an Azure Virtual Machine (VM), there’s a cost to keep it online. When used intermittently – for example during batch processes – this is inefficient for costs as you’re paying for the compute you don’t need. One way to alleviate this […]
Last week we looked at calling a Data Factory Pipeline from a Logic App. This week I thought we’d balance it out by taking a look at calling a Logic App from an Azure Data Factory (ADF) Pipeline. When building the Logic App last week we had to create our own polling mechanism to check for […]
When building Azure Logic Apps we can use the Azure Data Factory connector to start a pipeline. However that action simply triggers a pipeline and doesn’t wait for it to finish. If your downstream logic depends on the output – for example to collect a file – this can cause issues. In this post I’ll demonstrate how […]
