Myujth

Is there a built-in cmdlet or some composition thereof that would allow me to start unzipping a file stream as each chunk is downloaded? I have a PowerShell script that needs to download a large (10 GB) file, and I have to wait until it is done right now before it starts expanding...

$wc = New-Object net.webclient
$wc.Downloadfile($appDataSnapshotUri, "%DataSnapshotFileName%.zip") # this can take some time

Expand-Archive -Path "%DataSnapshotFileName%.zip" -DestinationPath Run # so can this

OK, turns out zip file doesn't need to be fully downloaded to be decompressed, you can compress/decompress streams. There is some built in capabilities in .Net for stream compression, but it will not work with zip archives. You can use SharpZipLib library for that:

Download .nupckg from https://www.nuget.org/packages/SharpZipLib/
Extract files to any folder. You'll need ICSharpCode.SharpZipLib.dll from lib/net45

Below is my simplified translation of their example:
https://github.com/icsharpcode/SharpZipLib/wiki/Zip-Samples#unpack-a-zip-using-zipinputstream-eg-for-unseekable-input-streams

Add-Type -Path ".ICSharpCode.SharpZipLib.dll"

$outFolder = ".unzip"

$wc = [System.Net.WebClient]::new()

$zipStream = $wc.OpenRead("http://gitlab/test/test1/raw/master/sample.zip")

$zipInputStream = [ICSharpCode.SharpZipLib.Zip.ZipInputStream]::New($zipStream)

$zipEntry = $zipInputStream.GetNextEntry()

$fileName = $zipEntry.Name

$buffer = New-Object byte 4096

$sw = [System.IO.File]::Create("$outFolder$fileName")

[ICSharpCode.SharpZipLib.Core.StreamUtils]::Copy($zipInputStream, $sw, $buffer)

$sw.Close()

It will only extract first entry, you can add a while loop it this sample works.

Here is a snippet with while loop to extract multiple files (put it after $zipEntry = $zipInputStream.GetNextEntry() on the example above):

While($zipEntry) 

$fileName = $zipEntry.Name

Write-Host $fileName

$buffer = New-Object byte 4096

$sw = [System.IO.File]::Create("$outFolder$fileName")

[ICSharpCode.SharpZipLib.Core.StreamUtils]::Copy($zipInputStream, $sw, $buffer)

$sw.Close()

$zipEntry = $zipInputStream.GetNextEntry()

Edit

Here is what I found to work...

Add-Type -Path ".ICSharpCode.SharpZipLib.dll"

$outFolder = "unzip"

$wc = [System.Net.WebClient]::new()

$zipStream = $wc.OpenRead("https://github.com/Esri/file-geodatabase-api/raw/master/FileGDB_API_1.5/FileGDB_API_1_5_VS2015.zip")

$zipInputStream = [ICSharpCode.SharpZipLib.Zip.ZipInputStream]::New($zipStream)

$zipEntry = $zipInputStream.GetNextEntry()

while($zipEntry) 

if (-Not($zipEntry.IsDirectory)) 
 $fileName = $zipEntry.Name

 $buffer = New-Object byte 4096

 $filePath = "$pwd$outFolder$fileName"
 $parentPath = "$filePath.."
 Write-Host $parentPath

 if (-Not (Test-Path $parentPath)) 
 New-Item -ItemType Directory $parentPath
 

 $sw = [System.IO.File]::Create("$pwd$outFolder$fileName")

 [ICSharpCode.SharpZipLib.Core.StreamUtils]::Copy($zipInputStream, $sw, $buffer)
 $sw.Close()



$zipEntry = $zipInputStream.GetNextEntry()

To expand on Mike Twc's answer, a script to do it with and without stream, and compare how long it takes:

$url = "yoururlhere"

function UnzipStream () 
 Write-Host "unzipping via stream"

 $stopwatch1 = [system.diagnostics.stopwatch]::StartNew()


 Add-Type -Path ".ICSharpCode.SharpZipLib.dll"

 $outFolder = "unzip-stream"

 $wc = [System.Net.WebClient]::new()

 $zipStream = $wc.OpenRead($url)

 $zipInputStream = [ICSharpCode.SharpZipLib.Zip.ZipInputStream]::New($zipStream)

 $zipEntry = $zipInputStream.GetNextEntry()

 while($zipEntry) 

 if (-Not($zipEntry.IsDirectory)) 
 $fileName = $zipEntry.Name

 $buffer = New-Object byte 4096

 $filePath = "$pwd$outFolder$fileName"
 $parentPath = "$filePath.."
 Write-Host $parentPath

 if (-Not (Test-Path $parentPath)) 
 New-Item -ItemType Directory $parentPath
 

 $sw = [System.IO.File]::Create("$pwd$outFolder$fileName")

 [ICSharpCode.SharpZipLib.Core.StreamUtils]::Copy($zipInputStream, $sw, $buffer)

 $sw.Close()
 

 $zipEntry = $zipInputStream.GetNextEntry()

 

 $stopwatch1.Stop()

 Write-Host "extraction took $($stopWatch1.ElapsedMilliseconds) millis with stream"


function UnzipWithoutStream() 

 Write-Host "Extracting without stream"

 $stopwatch2 = [system.diagnostics.stopwatch]::StartNew()
 $outFolder2 = "unzip-normal"

 $wc2 = New-Object System.Net.WebClient
 $wc2.DownloadFile($url, "$pwddownload.zip")

 $of2 = New-Item -ItemType Directory $outFolder2

 Expand-Archive -Path "download.zip" -DestinationPath $of2.FullName

 $stopwatch2.Stop()

 Write-Host "extraction took $($stopWatch2.ElapsedMilliseconds) millis without stream"


UnzipStream
UnzipWithoutStream