Last week we looked at the medallion architecture so I wanted to follow up by diving a little deeper into the Bronze layer by taking a practical example to compare options for retrieving data. Fabric provides a number of ways to bring data into the platform. Here I’ll sample 4 specifically: For this example I’ll use NHS monthly […]
Medallion architecture is the go-to for handling analytical data, particularly with the prominence of lakehouses in Databricks and Fabric. Whilst the layers are well defined, their definitions don’t tell you where particular tasks belong. So here I wanted to present my view from a more practical perspective of what goes into each layer. Layering Let’s start by […]
When ingesting files in SSIS via Flat File Connections, a consistent format is key. Sometimes that isn’t the case. Here we’ll look at an example where the carriage return (CR, \r) may or may not be included in the file. The problem A file format can change as easy as tweaking the file in an editor […]
When handing data we can make use of temporary tables to aid with separation or performance. However, they don’t always play nice with Integration Services packages. If we set a source to call a procedure returning the contents of a temporary table we’ll see an error like below: The process is unable to determine the […]
Integration workflows will typically involve handling files. As part of that we’ll need to move them around, for example moving into an archive directory. The File System Task component can be used for a variety of operations such as creating directories or copying files. Here we’re going to look at two specific operations to help […]
