Tag: Data Engineering

Categories
Fabric

Thinking in SQL, Working in Spark

I’ve spent years shaping data with SQL Server, however after pulling at the threads of Fabric I’m opening notebooks and finding PySpark. At first glance the difference is stark, but it’s not quite the dramatic shift it appears. If you’re not familiar, let’s look at what’s very similar, and where the true differences are. Same data, different […]

Categories
SSIS

Archiving Files with Move and Rename in SSIS

Integration workflows will typically involve handling files. As part of that we’ll need to move them around, for example moving into an archive directory. The File System Task component can be used for a variety of operations such as creating directories or copying files. Here we’re going to look at two specific operations to help […]