[SOLVED] Dart Flutter using Curl getting "Unexpected Extension Byte" error when using >> to append to file on MacOS

Issue

I think I'm misunderstanding something, formatting of the command maybe, but when I change this from using --output to using >> so that it appends to rather than just overwrites, like this:

var result = await Process.run('curl', ['--verbose', '--range', '$curStart-$curEnd','$baseURL$filename','--output', '$tempPath/$storedName']);

to this

var result = await Process.run('curl', ['--verbose', '--range', '$curStart-$curEnd','$baseURL$filename','>>', '$tempPath/$storedName']);

I get this error

[ERROR:flutter/lib/ui/ui_dart_state.cc(199)] Unhandled Exception: FormatException: Unexpected extension byte (at offset 1336)
#0      _Utf8Decoder.convertChunked (dart:convert-patch/convert_patch.dart:1891:7)
#1      _Utf8ConversionSink.addSlice (dart:convert/string_conversion.dart:314:28)
#2      _Utf8ConversionSink.add (dart:convert/string_conversion.dart:310:5)
#3      _ConverterStreamEventSink.add (dart:convert/chunked_conversion.dart:72:18)
#4      _SinkTransformerStreamSubscription._handleData (dart:async/stream_transformers.dart:111:24)
#5      _rootRunUnary (dart:async/zone.dart:1362:47)
#6      _CustomZone.runUnary (dart:async/zone.dart:1265:19)
#7      _CustomZone.runUnaryGuarded (dart:async/zone.dart:1170:7)
#8      _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:341:11)
#9      _BufferingStreamSubscription._add (dart:async/stream_impl.dart:271:7)
#10     _SyncStreamControllerDispatch._sendData (dart:async/stream_controller.dart:733:19)
#11     _StreamController._add (dart:async/stream_controller.dart:607:7)
#12     _StreamController.add (dart:async/stream_controller.dart:554:5)
#13     _Socket._onData (dart:io-patch/socket_patch.dart:2160:41)
#14     _rootRunUnary (dart:async/zone.dart:1370:13)
#15     _CustomZone.runUnary (dart:async/zone.dart:1265:19)
#16     _CustomZone.runUnaryGuarded (dart:async/zone.dart:1170:7)
#17     _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:341:11)
#18     _BufferingStreamSubscription._add (dart:async/stream_impl.dart:271:7)
#19     _SyncStreamControllerDispatch._sendData (dart:async/stream_controller.dart:733:19)
#20     _StreamController._add (dart:async/stream_controller.dart:607:7)
#21     _StreamController.add (dart:async/stream_controller.dart:554:5)
#22     new _RawSocket.<anonymous closure> (dart:io-patch/socket_patch.dart:1696:33)
#23     _NativeSocket.issueReadEvent.issue (dart:io-patch/socket_patch.dart:1208:14)
#24     _microtaskLoop (dart:async/schedule_microtask.dart:40:21)
#25     _startMicrotaskLoop (dart:async/schedule_microtask.dart:49:5)

But if I do the same call directly on the terminal like this

curl https://file I'm requesting >> `path/file I'm storing`

Solution

>> is a shell redirection. It isn't recognized or honored by curl -- it's only recognized by a shell, as an instruction about steps to follow before starting curl.

Hopefully, Dart will have a sufficiently powerful standard library to do that redirection natively -- for example, in Python, you pass stdout = open(f'{tempPath}/{storedName}', 'a') as an argument to subprocess.Popen to perform that redirection.

However, if dart doesn't give you a way to do that, the fallback is to start a shell:

var result = await Process.run('sh', [
  '-c', 'exec "\$@" >>"\$0"', '$tempPath/$storedName',
  'curl',
  '--verbose',
  '--range', '$curStart-$curEnd',
  '$baseURL$filename'
]);

dart is not a language I'm well familiar with, so if the literal syntax needs to be fixed up, please do that -- the exact string passed to the shell as the argument after the -c should be exec "$@" >>"$0". ("$0" refers to the position-0 argument, conventionally the name of the program being run but we're abusing it to store the name of the output file; "$@" refers to all arguments from $1 and onward). Note that >> can't be passed as part of a parameter value; the shell must parse it as code, rather than receiving it as data, for it to work as a redirection operator.

Here, we tell dart to start a shell directed to run all its arguments (implicitly, except $0) with output redirected to $0. That shell then starts curl -- and because of the exec, it runs curl in its existing PID rather than starting a subprocess, so the parent process can directly monitor and signal the copy of curl, instead of the shell that launched it.

Answered By - Charles Duffy