Last week we looked at the medallion architecture so I wanted to follow up by diving a little deeper into the Bronze layer by taking a practical example to compare options for retrieving data. Fabric provides a number of ways to bring data into the platform. Here I’ll sample 4 specifically: For this example I’ll use NHS monthly […]
Medallion architecture is the go-to for handling analytical data, particularly with the prominence of lakehouses in Databricks and Fabric. Whilst the layers are well defined, their definitions don’t tell you where particular tasks belong. So here I wanted to present my view from a more practical perspective of what goes into each layer. Layering Let’s start by […]
We recently dived into Spark and saw how DataFrames use lazy evaluation to defer execution until needed. This is useful for defining segments of logic to be reused, but if this causes large, complex calculations to execute multiple times, there may be a better way. Lazy evaluation Let’s take an example with some sales data: The display triggers execution […]
This month’s invitation from Marlon is sure to bring out the war stories as he asks us to share our most memorable SQL Server outage. I enjoy a good war story. But no, this isn’t the time Azure blindsided me, when service accounts were locked, or when I left a transaction open and halted a production system (easily fixed […]
I’ve spent years shaping data with SQL Server, however after pulling at the threads of Fabric I’m opening notebooks and finding PySpark. At first glance the difference is stark, but it’s not quite the dramatic shift it appears. If you’re not familiar, let’s look at what’s very similar, and where the true differences are. Same data, different […]
I’ve had Fabric gnawing at me for too long. Whilst I’ve dabbled in a couple of areas, I haven’t taken the time to explore the breadth of what Microsoft’s flagship data platform offers. That is, until I recently took the plunge. For those who may be in the same position, I wanted to share the […]
Date handling in SQL Server tends to accumulate tried and trusted combinations of DATEPART(), DATEADD(), and DATEDIFF() – with nested variations. The challenge with these isn’t raw performance, but more with conveying intent and readability. So let’s look at some simpler patterns to try and avoid some of these and be clearer with what we’re […]
For August’s T-SQL Tuesday ask, Jeff has thrown temp tables to the wolves. Does the community like or loathe their usage? I’ll predict a few ‘it depends’ responses, and I lean towards Jeff’s position that they’re more of the exception than the rule. I’d argue there’s one particular scenario where this exception becomes exceptional (heh). Building optimisation boundaries […]
Computed columns allow us to bake logic into our schema. I find particular use for these when reviewing or optimising established solutions where you spot the quirky ways the data is being used. Here I want to demonstrate 3 symptoms and show how different implementations of computed columns solve them. Baking the expression Common expressions […]
Fragmented data estates usually share a trait – there’s no consistent representation of core business entities. It’s a common thread I’ve seen – system-centric entities and attributes, then another solution which recreates these in its own unique way. Different systems talking different languages make integrations messy and bespoke. It’s a technical mess, and it doesn’t […]
