In Synapse SQL Pools, Hash distributions allow us to somewhat control data distribution. Whilst this can deliver performance in the right scenarios, I noted last week that these tables should be monitored to avoid skew. In this post I want to demonstrate how to identify the skew, and why a different distribution model may be […]
When designing tables in a Synapse Dedicated SQL Pool, I initially focussed on the ‘SQL’ and not so much the ‘pool’. How tables are stored is fundamentally different to other Azure SQL offerings, so it’s easy to get tripped up and impact efficiency and performance. Let’s see how the SQL pools work differently and how […]
Retries are a useful part of resilient pipelines. However, retries aren’t created equal, so I want to dive into recent changes to pipeline activity retries and the benefits they bring. The issue Failures happen, so we have retries to have another chance (or few) at completing the activity. They look something like this: These fixed […]
Pipeline sequences are a common orchestration pattern, and chaining them allows us to decompose processes into manageable chunks. Here I want to look at 3 ways to achieve this chaining in Fabric and how they compare. Invoke pipeline The default option is the native Invoke Pipeline activity. This allows us to call pipelines wherever they […]
When combining and curating data in your Silver layer it’s not uncommon to find entries don’t always fit quite right. It could be missing references, incomplete data, or broken validation rules. Should that bad data be allowed into your carefully crafted Silver layer? And if not, where should that data go? Let’s look at some options for […]
Last week we looked at the medallion architecture so I wanted to follow up by diving a little deeper into the Bronze layer by taking a practical example to compare options for retrieving data. Fabric provides a number of ways to bring data into the platform. Here I’ll sample 4 specifically: For this example I’ll use NHS monthly […]
Medallion architecture is the go-to for handling analytical data, particularly with the prominence of lakehouses in Databricks and Fabric. Whilst the layers are well defined, their definitions don’t tell you where particular tasks belong. So here I wanted to present my view from a more practical perspective of what goes into each layer. Layering Let’s start by […]
We recently dived into Spark and saw how DataFrames use lazy evaluation to defer execution until needed. This is useful for defining segments of logic to be reused, but if this causes large, complex calculations to execute multiple times, there may be a better way. Lazy evaluation Let’s take an example with some sales data: The display triggers execution […]
This month’s invitation from Marlon is sure to bring out the war stories as he asks us to share our most memorable SQL Server outage. I enjoy a good war story. But no, this isn’t the time Azure blindsided me, when service accounts were locked, or when I left a transaction open and halted a production system (easily fixed […]
I’ve spent years shaping data with SQL Server, however after pulling at the threads of Fabric I’m opening notebooks and finding PySpark. At first glance the difference is stark, but it’s not quite the dramatic shift it appears. If you’re not familiar, let’s look at what’s very similar, and where the true differences are. Same data, different […]
