Data Engineer and Nerd
January is a great time of year to raise your head and take a look around. Looking back at what has been, where we’re going, and the technical and mental debt which needs a clear out. I skipped this Year in Review last year due to other commitments, so there’s a little more ground to […]
It’s not the usual time of year to think about Halloween, but listening to Erik Darling and Kendra Little recently on the Dear SQL DBA podcast they mentioned a completely new topic to me: Halloween protection. No, it doesn’t prevent horrors in your code, or shops putting out festive decorations in October. It’s actually much more interesting and thought it was worth […]
Last week we looked at how expressions are evaluated with the ISNULL and COALESCE functions. Whilst we’re in the area it’s worth running through how data types are selected for them too. It might have implications with how you implement them. The way these functions choose which data type to return differs, so they aren’t direct swap outs for each other, […]
When eliminating NULL values with SQL Server queries we typically reach for ISNULL or COALESCE to do the job. Generally speaking they’ll provide equivalent results, but they work in different ways. If you’re dealing with logic more complex than ISNULL(ColA, ColB) – for example using a function or subquery as part of the expression – then you might be in for a surprise. […]
This month’s invitation from Mike asks us to write a couple of notes – one to our past self, and one from our future self – focused on what we were concerned about, and what we should be concerned about. I love it, and it’s been itching away at me all week. So onto the messages. First from 2025 back to 2020 Andy […]
Not all pipelines in Azure have native options for sending emails. The lucky Fabric folks have an Office 365 connection ready to rock. Data Factory and Synapse don’t have email capability built-in and need another option to send messages. So for everyone who wants more emails for their inbox rules to handle (delete), we’ll build and call […]
If you need to create a copy of a table in another database, the ‘Import Data’ option may seem convenient. If you’ve used this method to copy to your dev environment and found things break, this post is for you. By letting the wizard create the destination table automatically, you might find your schema is […]
Slow queries are one thing, but under heavy pressure, your SQL Servers might see failed connections or slow responses. When the server isn’t responding we want a back-door we can crack open. In SQL Server, this comes in the form of a Dedicated Administrator Connection (DAC). In this post I want to look at what […]
If you have a Self-Hosted Integration Runtime (SHIR, or IR for short here) on an Azure Virtual Machine (VM), there’s a cost to keep it online. When used intermittently – for example during batch processes – this is inefficient for costs as you’re paying for the compute you don’t need. One way to alleviate this […]
Last week we looked at calling a Data Factory Pipeline from a Logic App. This week I thought we’d balance it out by taking a look at calling a Logic App from an Azure Data Factory (ADF) Pipeline. When building the Logic App last week we had to create our own polling mechanism to check for […]
Andy Brownsword 