Data Engineer and Nerd
When eliminating NULL values with SQL Server queries we typically reach for ISNULL or COALESCE to do the job. Generally speaking they’ll provide equivalent results, but they work in different ways. If you’re dealing with logic more complex than ISNULL(ColA, ColB) – for example using a function or subquery as part of the expression – then you might be in for a surprise. […]
This month’s invitation from Mike asks us to write a couple of notes – one to our past self, and one from our future self – focused on what we were concerned about, and what we should be concerned about. I love it, and it’s been itching away at me all week. So onto the messages. First from 2025 back to 2020 Andy […]
Not all pipelines in Azure have native options for sending emails. The lucky Fabric folks have an Office 365 connection ready to rock. Data Factory and Synapse don’t have email capability built-in and need another option to send messages. So for everyone who wants more emails for their inbox rules to handle (delete), we’ll build and call […]
If you need to create a copy of a table in another database, the ‘Import Data’ option may seem convenient. If you’ve used this method to copy to your dev environment and found things break, this post is for you. By letting the wizard create the destination table automatically, you might find your schema is […]
Slow queries are one thing, but under heavy pressure, your SQL Servers might see failed connections or slow responses. When the server isn’t responding we want a back-door we can crack open. In SQL Server, this comes in the form of a Dedicated Administrator Connection (DAC). In this post I want to look at what […]
If you have a Self-Hosted Integration Runtime (SHIR, or IR for short here) on an Azure Virtual Machine (VM), there’s a cost to keep it online. When used intermittently – for example during batch processes – this is inefficient for costs as you’re paying for the compute you don’t need. One way to alleviate this […]
Last week we looked at calling a Data Factory Pipeline from a Logic App. This week I thought we’d balance it out by taking a look at calling a Logic App from an Azure Data Factory (ADF) Pipeline. When building the Logic App last week we had to create our own polling mechanism to check for […]
When building Azure Logic Apps we can use the Azure Data Factory connector to start a pipeline. However that action simply triggers a pipeline and doesn’t wait for it to finish. If your downstream logic depends on the output – for example to collect a file – this can cause issues. In this post I’ll demonstrate how […]
This month’s invitation from Steve asks us about our favourite string parsing routines. It conveniently popped up as I was trying out the new Regular Expression (Regex) functions in SQL Server 2025. I adore them for handling string validation and parsing, which I’ve historically done in .Net or PowerShell. Now that we’re on the cusp of adoption […]
String parsing is a regular occurrence when it comes to data engineering. Splitting strings, converting, combining – all in an effort to bring some order. Sometimes we need a more powerful tool, and that would be Regular Expressions (Regex). Regular expressions are something I have a fondness for under the right conditions, and with them being introduced […]
Andy Brownsword 