Pipeline sequences are a common orchestration pattern, and chaining them allows us to decompose processes into manageable chunks. Here I want to look at 3 ways to achieve this chaining in Fabric and how they compare. Invoke pipeline The default option is the native Invoke Pipeline activity. This allows us to call pipelines wherever they […]
When combining and curating data in your Silver layer it’s not uncommon to find entries don’t always fit quite right. It could be missing references, incomplete data, or broken validation rules. Should that bad data be allowed into your carefully crafted Silver layer? And if not, where should that data go? Let’s look at some options for […]
Last week we looked at the medallion architecture so I wanted to follow up by diving a little deeper into the Bronze layer by taking a practical example to compare options for retrieving data. Fabric provides a number of ways to bring data into the platform. Here I’ll sample 4 specifically: For this example I’ll use NHS monthly […]
Medallion architecture is the go-to for handling analytical data, particularly with the prominence of lakehouses in Databricks and Fabric. Whilst the layers are well defined, their definitions don’t tell you where particular tasks belong. So here I wanted to present my view from a more practical perspective of what goes into each layer. Layering Let’s start by […]
We recently dived into Spark and saw how DataFrames use lazy evaluation to defer execution until needed. This is useful for defining segments of logic to be reused, but if this causes large, complex calculations to execute multiple times, there may be a better way. Lazy evaluation Let’s take an example with some sales data: The display triggers execution […]
I’ve spent years shaping data with SQL Server, however after pulling at the threads of Fabric I’m opening notebooks and finding PySpark. At first glance the difference is stark, but it’s not quite the dramatic shift it appears. If you’re not familiar, let’s look at what’s very similar, and where the true differences are. Same data, different […]
I’ve had Fabric gnawing at me for too long. Whilst I’ve dabbled in a couple of areas, I haven’t taken the time to explore the breadth of what Microsoft’s flagship data platform offers. That is, until I recently took the plunge. For those who may be in the same position, I wanted to share the […]
When developing Power BI models, we don’t always want to refresh the entire dataset before we start working with the data. We can speed up the refresh – and therefore our development – by reducing the volume we’re working with. Here we’ll look at a couple of ways to use parameters to adapt production-ready models to reduce […]
Last week we looked at utilising Teams to make notifications more collaborative. This week I want to show how we can add more visual and interactive elements by adopting Adaptive Cards. What are Adaptive Cards? Adaptive Cards are structured messages which can be used across various Microsoft services as alternatives to plain text. They use JSON […]
Whether running Data Factory, Synapse, or Fabric pipelines, things go wrong – and the de facto response is to send an email. We’ve looked at sending emails from pipelines before, but at scale they can become noise and are easy to ignore. A more effective option is to surface alerts where collaboration already exists, such as Teams. […]
