[SOLVED] Curl Powershell windows 10 slower than command prompt why?

Issue

Just a pretty stand curl command call an S3 end point for download using all default values. On a mac, or on a PC using command line I get 103MBsec if cached on cdn and 80mbsec otherwise. Same command, same bucket, same object, using "curl.exe" and I get 1MBSec when call through powershell. I guess powershell does something different that make it's totally slow? I tried using newest curl binary but still the same. I guess I am misunderstanding what powershell is doing when I use a curl command

curl.exe yourfileonS3 >> output.bin

Solution

Update:

• PowerShell (Core) v7.4+ now does support raw byte handling with external programs - see this answer.

• The following therefore applies only to Windows PowerShell and PowerShell (Core) v7.3-

---

To complement briantist's helpful answer:

• In PowerShell, the redirection operators > and >> are in effect aliases of Out-File and Out-File -Append.

• > and >> are therefore not mere byte-stream conduits, and, in fact, Windows PowerShell and PowerShell (Core) up to v7.3 do not support sending raw byte output to a file.

  • Instead, PowerShell invariably decodes output from any external program as text ([string] instances), based on the character encoding reported by [Console]::OutputEncoding] and then, on saving to the target file with Out-File (possibly via > / >>), re-encodes these strings, using that cmdlet's default character encoding (unless overridden with -Encoding in an explicit Out-File call).

  • Not only does this not preserve the external program's raw byte output, it adds significant overhead.

To get raw byte processing, call cmd.exe^[1] and use its redirection operators:

cmd /c 'curl.exe yourfileonS3 >> output.bin'

See this answer for more information.

---

^{[1] On Unix-like platforms, use sh -c 'curl yourfileonS3 >> output.bin'}

Answered By - mklement0

Answer Checked By - Gilberto Lyons (WPSolving Admin)