Robust, Asynchronous, Throttled file transfers using PowerShell

(Shamelessly copied from an article I wrote for work)

Sometimes you have a bunch of My Little Pony episode files in MP4 format on a box, and you want to transfer them across the network to another box. Except these boxes are in Azure with flaky VNet links that don't like it when a single box saps all the bandwidth. Oh, and the MP4 files are actually large SQL database backup files.

Powershell can utilise the BITS (Background Intelligent Transfer Service) which is able to do useful stuff like kick off file transfers asynchronously, limit bandwidth usage and so on. So, you could start a copy, and subsequently you get logged off the box, and then it reboots for no good reason. The BITS job will still continue to run when it left off. The Priority flag set to Low will ensure that only idle bandwidth is used.

Start-BitsTransfer -Source '.\SQLDB.bak' -Destination '\\FileServer\Backups\' -Asynchronous -Priority Low

After you've done this, you can check on the status of the transfers running by using the following:

$transfers = Get-BitsTransfer

(or to retrieve transfers for all users on the system...)

$transfers = Get-BitsTransfer -AllUsers

The $transfers object will contain an array of transfers in progress - you can then examine each one and look at the .BytesTransferred property to see how far through the transfer it is. The really cool thing about this is that this object is dynamic - you don't need to keep re-issuing the above command to get the latest byte count:

PS> $transfers = Get-BitsTransfer
PS> $transfers[0].BytesTransferred
5581570048
PS> $transfers[0].BytesTransferred
5591007232
PS> $transfers[0].BytesTransferred
5597298688
PS> $transfers[0].BytesTransferred
5600444416

Once the Get-BitsTransfer cmdlet indicates that the job has finished transferring (the status will be 'Transferred'), you need to finalise it by calling Complete-BitsTransfer on the transfer objects that you're interested in. This will complete all transfers:

Get-BitsTransfer | Complete-BitsTransfer