Allow long arguments to be provided via a file instead of only on the command line

[DanTup]

On Windows, there is a limit to the size of a command line (executable + arguments). It's easy to hit this limit with some operations like asking the IDE to re-run all test files that had failing tests from the last run, because each test file needs to be listed specifically:

One option to fix this for dart test is to support an argument like --tests-from-file testlist.txt, however this only solves the issue for that specific case, but it would be nice to solve this more generally.

@jakemac53 noted that Bazel supports a final argument of @filename that can be used to read arguments from a file which seems like a nice convention to follow (and I started implementing at dart-lang/test#2485), and this is also implemented in Dart compute_kernel.dart here.

When trying to add the equivalent to flutter test, it turned out to be easier to support for the whole flutter command because of how the CommandRunner works (I started this at flutter/flutter#167272). If doing this for all of flutter it would be nice if we also supported it for all of dart (instead of only pkg:test).

@bkonyi had some questions that would be worth discussing if we're going to implement this more generally across all tools/commands (and I added a few):

• Are we concerned about legit args that start with @ that aren't this file?
• Should we support replacing args-from-file that aren't in the final position of the command?
• Exactly what should the format of the file be?
  • Bazel seems to support a few formats but the Dart code here seems to just assume raw args on individual lines (I presume this makes it impossible to provide newlines in the values)

I don't personally have any real preference for what the solution is, my main goal is to be able to resolve bugs like the one shown up-top. I do think a general solution implemented once in dart ... and flutter ... would be better than having multiple implementations in individual sub-commands though.

[bwilkerson]

I asked Gemini that question just out of curiosity. The full answer is below, but based on it I would say that we shouldn't need to worry about using @.

We do need to decide whether arguments on the command-line override arguments from the file or whether they're all treated equally. I like the idea of being able to override what's in the file, but that might not be the right answer.

I'm not sure what advantage there would be of using spaces to separate arguments, but unless there are some I'd probably argue for one argument per line, given that the whole point of the file is to contain long lists of arguments (which would be hard to edit if they can't be broken across lines).

Probably unnecessary to say, but if we do provide this support we should add it to the args package so that it's available everywhere.

---

Q:
Is there a widely used convention for providing a file containing additional command line arguments to a command-line tool?

A:
While there isn't one universally enforced standard, there are common practices and conventions that command-line tools employ for handling additional arguments from files. Here's a breakdown:

Common Practices:

• @ symbol:

  • A frequent convention involves using the "@" symbol followed by the filename. When a command-line tool encounters "@filename," it reads the file and treats its contents as additional command-line arguments.
  • This is seen in some tools where for example "@myarguments.txt" would cause the program to read the arguments contained within "myarguments.txt".

• Configuration Files:

  • Many tools support configuration files, often in formats like:
    • Plain text files (where arguments are listed one per line or space-separated).
    • INI files.
    • JSON or YAML files.
  • These files allow for complex configurations and are often used when numerous or intricate arguments are needed.

• Environment Variables:

  • While not strictly "files," environment variables serve a similar purpose, allowing users to store and retrieve values that influence a program's behavior.

• Argument Parsing Libraries:

  • Libraries like Python's argparse provide functionality to handle various argument parsing scenarios, including the potential to create custom logic for reading arguments from files. This makes it easier for developers to implement these features.

Key Considerations:

• Syntax:

  • The syntax for how arguments are interpreted from a file can vary. Some tools expect one argument per line, while others allow space-separated arguments.

• Comments:

  • Some tools allow comments within argument files, often using characters like "#" to denote lines that should be ignored.

• Precedence:

  • It's crucial for tools to define the precedence of arguments provided via files versus those given directly on the command line. Typically, command-line arguments override those from files.

In essence, while a rigid standard is lacking, the "@" symbol and the use of configuration files are prevalent methods for supplying additional command-line arguments.

[DanTup]

We do need to decide whether arguments on the command-line override arguments from the file or whether they're all treated equally. I like the idea of being able to override what's in the file, but that might not be the right answer.

I like the idea of just saying that whatever is read from the file replaces the @file in-place and therefore all behaviour is the same as if you'd done it manually. Having a notion of overriding in the substitution could get confusing for args that can be supplied multiple times (are both supplied, or does one replace the other?).

I'm not sure what advantage there would be of using spaces to separate arguments, but unless there are some I'd probably argue for one argument per line, given that the whole point of the file is to contain long lists of arguments (which would be hard to edit if they can't be broken across lines).

Yeah, I think supporting spaces would also require args to be quoted/escaped if they contain spaces, which makes things more complicated on both sides (particularly as escaping across shells/platforms can differ and is a bit of a minefield as flutter/flutter#84270 / #59604 have shown 🙃). I think with line-separated we could probably just forbid args that contain newlines and therefore have no escaping at all.

[rmacnak-google]

gcc/clang/msvc call these response files.

[dcharkes]

Yes, this would be great! Being able to pass a file also was requested in dart-lang/native#39.

TL;DR: Use JSON as the format instead of plain text.

Format

Terminals support spaces and newlines with quotes and interpolation:

your_command "foo bar"

your_command $'line1\nline2'

your_command "line1
line2"

1. Plain text & is the argument separator

Verbatim what one would write in the command-line invocation but no terminal-like string interpolation.

This doesn't really work because it wouldn't support spaces inside arguments. Every space would be interpreted as a new String in List<String>

Pro:

• Verbatim how one would pass arguments on command-line.

Con:

• Doesn't work for spaces and new-lines.

2. Plain text & \n is the argument separator

@DanTup's suggestion IIRC.

dart main.dart @file arguments.txt

arguments.txt:

foo
bar baz

Would lead to ["foo", "bar baz"] as arguments.

New lines are the argument separator, spaces are spaces in arguments.

dart main.dart foo "bar baz"

I think with line-separated we could probably just forbid args that contain newlines and therefore have no escaping at all.

I believe this might end up biting us in the end. Users probably shouldn't be using new lines in command-line arguments, but they might. I'm assuming people are going to write abstractions over this mechanism, and it's likely someone will try to pass newlines in at some point.

Pro:

• Works for spaces.

Con:

• Not verbatim how one would pass arguments on command-line.
• Doesn't work for new-lines.

3. Plain text & terminal-like evaluation for strings/spaces/newlines

I like the idea of just saying that whatever is read from the file replaces the @file in-place and therefore all behavior is the same as if you'd done it manually.

We could try to support verbatim what's in the file is interpreted as if one was in the terminal.

However, this means we'll now need to duplicate the bash behavior inside Dart/Flutter. (We'd probably rule out other behavior that is not simply strings related such as $(pwd).)

Pro:

• Verbatim how one would pass arguments on command-line.
• Works for spaces and new-lines.

Con:

• Implementation complexity in Dart/Flutter.

4. Structured input: JSON / yaml

If the goal is pass something into List<String> args, then a way to completely skip parsing questions w.r.t. spaces and tabs is to use a format for which this is already properly defined, JSON:

[
  "--arg1",
  "my_value",
  "--arg2",
  "happens to contain spaces",
  "--arg3",
  "another\nargument\nwith\nmultiple\nnewlines"
]

Or if we expect these files to be edited by humans frequently, we could consider supporting YAML. (YAML would also support comments, JSON does not.)

Pro:

• Works for spaces and new-lines.
• No ambiguity for how parsing is done.

Con:

• Not verbatim how one would pass arguments on command-line.

I'm leaning towards option 4, which would rule out any ambiguity for how spaces and new-lines are parsed.

Issues with spaces in arguments and escaping are notoriously painful. Process.start have the runInShell option which changes how your arguments are escaped/parsed. And I've had weird things with spaces happen multiple times. Choosing a format in which the parsing for new-lines and spaces is well-understood avoids such pain.

@file or @filename

@file is shorter than @filename, so I prefer @file.

[DanTup]

I think I like the idea of a JSON string array.. it's a little less simple than verbatim lines in a text file, but it avoids the newline issue and makes the format more obvious/less ambiguous (probably requiring less documentation). I'm less sure about YAML.. having comments is nice, but the format is less simple (particularly to parse) and it'd be nice to have something that's easy to use across Dart/non-Dart (in this case, Dart-Code is TypeScript).

@file or @filename
@file is shorter than @filename, so I prefer @file.

I'm not sure I understand this - the idea was for the part after the @ to be an actual filename/path, so you would write something like:

dart test @foo/myargs.txt

So you wouldn't literally write @file or @filename. Of course if your interpretation was to do @file foo/myargs.txt that would work too, but I'm not sure if there's an additional value (besides reducing the scope of restricted legit args from anything starting with @ to only @file).

While searching for "repsonse files" as noted above, I found this:

mozilla/sccache#43

The options read are inserted in place of the original https://github.com/file option. If file does not exist, or cannot be read, then the option will be treated literally, and not removed.

So I guess there's a question of whether to do something like that, or just fail with an error if the file is missing/invalid. My feeling is that rejecting is better (assuming a missing file is because something is wrong, and not that it is a legit @ argument), but I can see an argument both ways.

[dcharkes]

dart test @foo/myargs.txt

Ah, I completely misunderstood that. Yeah dart test @path/to/file.json sgtm!

fail with an error if the file is missing [...] My feeling is that rejecting is better

+1

[jakemac53]

While JSON would be simpler on the edge cases we have existing use cases which follow the newline separated args pattern. Newlines in arguments are technically a thing but exceedingly rare.

Bazel also does not support Json as a format, and this is one of our primary use cases. Technically we could create our own params files using JSON but I think its a strong signal that this support is not necessary.

[natebosch]

+1 for using the same newline separated format as bazel if we are using the same @filename argument pattern.

[DanTup]

It sounds like there's a preference towards text instead of JSON. Is this enough of a consensus to implement something like this?

• a plain text file with one argument per line, literally (no kind of escaping - which means embedded newlines are not supported)
• support multiple @path/filename args in any position that will be replaced in-position with the args from the file
• the filename/path is relative to current working directory
• it's the responsibility of the caller to clean up the file (the processing parsing the arg will not delete it)

Some questions I'm not sure about:

1. What do we do if we get @foo and there is no file named foo (or we otherwise fail to read it)?
   a. throw an error (this will break anything with legit @ args)
   b. leave the arg as-is (this supports other @ args as long as they aren't real filenames we'd find)
2. Should we specific the file encoding? (for ex. UTF8 without BOM?)
3. Where is this best implemented?
   a. pkg:dartdev in bin/dartdev.dart main()
   b. pkg:dartdev in lib/dartdev.dart runDartdev()
   c. pkg:args? (note: DartdevRunner does some contains() checks on the args and parses VM experiments before it ever gets to pkg:args so this may either need some refactoring, or wouldn't support all args in the file)
   d. somewhere else (are there any paths that don't go through pkg:dartdev that should support this)?

[dcharkes]

It sounds like there's a preference towards text instead of JSON.

If everyone else prefers text, and no support for newlines, that's fine for me.

cc eco system @devoncarew @mosuem and dartdev flutter_tools @bkonyi in case they have an opinion.

RE 3: If it is implemented in dartdev, then it should also be implemented in flutter_tools. I'd say it should be implemented in dartdev, not package:args. And preferably as early in the code as possible. I'd consider it as "before any other code is running". E.g. whatever gets the List<String> args from main would only see the expanded version.

In addition to dartdev and flutter_tools it would be nice if we also implement it in the Dart VM. E.g. dart --disable-dart-dev @file/path.txt should also work. This must be implemented in runtime/bin/main_impl.cc most likely.

[bkonyi]

Some questions I'm not sure about:

1. What do we do if we get @foo and there is no file named foo (or we otherwise fail to read it)?
   a. throw an error (this will break anything with legit @ args)
   b. leave the arg as-is (this supports other @ args as long as they aren't real filenames we'd find)

I think B) is the best approach here.

2. Should we specific the file encoding? (for ex. UTF8 without BOM?)

We need to be able to consistently decode the contents of the file to do anything, so we should be explicit about what formats are supported and try and be consistent with what we support in other situations where the user can provide a file argument to an option.

3. Where is this best implemented?> a. pkg:dartdev in bin/dartdev.dart main()
   b. pkg:dartdev in lib/dartdev.dart runDartdev()
   c. pkg:args? (note: DartdevRunner does some contains() checks on the args and parses VM experiments before it ever gets to pkg:args so this may either need some refactoring, or wouldn't support all args in the file)
   d. somewhere else (are there any paths that don't go through pkg:dartdev that should support this)?

Ideally we'd add this support to package:args, even if this behavior needs to be explicitly enabled, as we'd get it for free across all of our tooling.

And preferably as early in the code as possible. I'd consider it as "before any other code is running". E.g. whatever gets the List args from main would only see the expanded version.

I'm not sure we should be modifying the contents of args as it's currently identical to what the user is providing via the CLI.

In addition to dartdev and flutter_tools it would be nice if we also implement it in the Dart VM. E.g. dart --disable-dart-dev @file/path.txt should also work. This must be implemented in runtime/bin/main_impl.cc most likely.

It'd be nice to support this for VM arguments, but I think we should leave processing of Dart arguments to the user program if we can.

[devoncarew]

Ideally we'd add this support to package:args, even if this behavior needs to be explicitly enabled, as we'd get it for free across all of our tooling.

adding it to package:args seems reasonable, esp. if there's some prior art / existing convention we can point to

[jakemac53]

Adding it to args sounds fine to me, but we might want it to be available as a standalone feature (ie: top level expandArgsFromFile function or something), to allow for things that want to be able to scrape/alter the arguments before passing them along to the parser.

And then I think it should be something you can opt in/out of for the ArgParser itself, whether it will do this expansion internally. I don't have a strong opinion on what the default should be, but opt in is less likely to cause any problems.

[munificent]

Putting it in args sounds reasonable to me. It will take a little care because I believe most of the args package is currently not coupled to dart:io or having access to the file system. (All it does is take in a list of strings and parse it.) I'm sure most users of the args package are using it on the command-line VM, but it may still be a breaking API change to have the main args library itself import dart:io.

[jakemac53]

Good point. We could use conditional imports and only support it when dart:io is available. It could either throw or just do nothing if its not available and an @<something> arg is passed.

[bwilkerson]

Another option would be to take a callback that can return the contents of the file given the 'path' after the @.

[jakemac53]

Another option would be to take a callback that can return the contents of the file given the 'path' after the @.

If its optional, then that SGTM. We should make the 99% case as easy as possible though and not require everybody to pass a callback that just uses dart:io.

[natebosch]

I'd first look at adding it in a separate library, potentially as an extension, so that authors explicitly import it in places with dart:io available.

[DanTup]

So are we suggesting a top-level function in a new library (which will be an empty stub when there's no dart:io):

import 'package:args/args_file.dart`;

void main(List<String> args) {
  args = expandArgsFiles(args);
  // ...
  var argResults = parser.parse(args);
}

And also a flag for ArgParser() that just does this automatically?

var parser = ArgParser(
  expandArgsFiles = true,
);


class ArgParser {
  ArgResults parse(Iterable<String> args) {
    if (expandArgsFiles) {
      args = expandArgsFiles(args);
    }

    // ...
  }
}

I'm still slightly unsure about "doing nothing" if an invalid/unreadable filename is passed.. it could make the error message if there are typos less useful (although, I think I expect this feature mostly to be used by tools and not being typed by users).

[natebosch]

I was imagining something like

import 'package:args/args_file.dart';

void main(List<String> args) {
  var argResults = parser.parseWithArgsFile(args);
}

A separate utility to expand into a List<String> to pass to parse would also work.

[sigurdm]

I like the idea.

In pub publish we want to call dart analyze with the specific list of files to analyze (so gitignored files don't get analyzed, but we worry about the length of the command-line.

One question: Are nested @filename expansions expanded recursively?

[DanTup]

One question: Are nested @filename expansions expanded recursively?

My assumption was that they are not (I don't think that's supported in some of the other cases noted above), so the strings are always literal. I don't have a preference though (it's simple enough to support, though we would have to watch for infinite loops).

[lrhn]

Good question from Sigurd. This is a feature, especially if added to package:args, so we should make sure we design it properly.

The absolute zero-smartness base feature would be: At the start of parsing command line arguments, if the last argument starts with @, and what follows denotes a file, read that file as a UTF-8 text file, split into lines using LineSplitter, and replace the final @-argument with that list.
That implies: No recursive expansion, no way to avoid the expansion other than adding a later argument.

Use- and edge-cases to consider:

• Is this specifically for providing files to the dart executable, or is it a general feature. (If it's in package:args, it's general.)
• What if someone wants the @ syntax for themselves? Can we provide a way to disable @, and/or make it accessible as a flag, like --args=path, --with-flags=path or similar.
• Is @ only allowed as the last argument, or can there be arguments after? (If it's files, you might want to make sure they're processed before another file that you provide explicitly, like foo run @preambles main.foo.)
• Can there be more than one @-entry? (No technical reason not to.)
• What if it occurs after a -- argument, should it still be expanded? (Probably not, so -- @foo is a way to avoid expanding to the contents of a foo file. So is @foo -- if we only expand the last argument.)
• What if a file contains a -- argument?
  • Should inhibit interpretation of flags in the file.
  • If the @path can occur in non-final position, does it also affect disable flags after the @path? (Probably not. If only allowed as last argument, that question is moot.)
• Can a file end in a flag whose value is not in the file? Say foo @file 42 where file ends with --with-count, so that the flag+value becomes equivalent to --with-count=42. Same if file2 starts with 42 in foo @file @file2.
• Can @ entries occur in @ files?
  • If so, are they relative to the CWD or the file? (The file!)
    • Can you provide a base location for resolving the top-level ones to package:args?
    • Should it be a URL or a package:path path?
  • If not, what happens if there is a @foo in a @-file? Included verbatim most likely.
• Can you escape @, if you absolutely positively need a command line argument that starts with @.
  • If so, how. Probably @@ or \\@ on the command line, and @@ or \@ in a file.
  • If not, what do you do? Put it in a file, but only if we don't have recursive expansion?
• How must @-files be encoded? Must it be UTF-8 encoded, latin-1, or ASCII-only? UTF-16 with BOM?
• Can line terminators be \r\n or only \n. (I'd allow both, and \r alone too, all the Unicode ASCII-only line terminator grapheme clusters, or anything that LineSplitter splits.)
• What does an empty line mean? Is it an empty argument or no argument?
• Should we allow comments, probably by starting a line with #? (Yes, please!)
  • If so, empty lines can be empty arguments, like "", if you didn't want that, add a #.
    • But then be damn certain that \r\n is parsed as one line break.
  • How do you then escape a leading #?
• Escaping: Maybe allow a line to start with with \, meaning that the next character is verbatim, even if it is @, # or \. So \ is an identity-escape, but only as the first character of a line, so you need \\ for an actual leading '\, but can use \@ and \# to escape significant characters at the start. No escapes later on lines.
• Or allow \n and \r to insert newlines into lines, if an argument needs to contain a newline.
  • Then \ is a general escape, and we need \\ to use it verbatim, everywhere on a line.
  • Or only allow \ before a newline to make the newline linteral (escaped as \\ before a newline), combining it with the next line joined by a line terminator, no matter what is on that next line. (Which line terminator? The one in the file or normalized to \n?)
• Is what's after @ a path or a relative URL, or both? (Aka: Accept \ on Windows? Also accept / on Windows? Allow schemes to fetch from HTTP? ... probably not.)
• You can use environment variables on the command line. Should we allow them in the file too?
  • Maybe --output=$DART_DIR/out.exe? Allow \$\w+ and \$\{[^}]*} and let them access the shell's environment.
• If you have @some/path, and some/path doesn't specify a file, what happens?
  • Retained verbatim is the "nice" behavior that doesn't affect people unaware of the feature, ... unless they happen to hit the name of a file. (Don't do foo annotations add @con file.foo on Windows, even if it works on Unixes.)
  • I'd prefer an error. If you want a command line argument starting with @, you must escape it.
  • (Probably not a problem if it's only allowed as the last argument, but you can still hit it accidentally.)
  • What if it denotes a directory?
  • What if it does denote a file, but it's not valid text? (What is valid text? See encoding above.)
• @ with nothing after. Will it try to read File(""), aka CWD, as a file? Is it an error?
• @"banana split". Should work, it's a file name with a space in it.
• In a file, @banana split should also work, no quoting required, it's a single argument per line.
• Should we trim @-file lines before parsing?
• Should we have a suggested extension for the file, like foo.args, maybe foo.files/foo.paths if it's only used for files, foo.flags if only used for flags. Or just leave it to the user, who'll likely use .txt. (It's not wrong.)
  • If we have a non-trivial format, someone may eventually want some kind of validation or syntax highlighting for it. I guess those people can decide on the extensions if we don't do anything.
• Allow package:args to be configured with user-provided functions to load files from paths, and read environment variables, which default to throwing if there is no access to dart:io, and using dart:io's File.readAsStringSync/Platform.environment otherwise.

A "full-featured" proposal would allow @path anywhere, and multiple times, in an argument list, and allow recursive expansion, have escapes to allow all lines and newlines to be part of the expansion.
I'd introduce an "argument iterator" which iterates through arguments, recursively through files pointed to by @path arguments, and which tracks occurrences of the -- for each list or file independently.

That is an iterator which has String get current and bool get flagsCurrentlyDisabled where the latter is set by seeing a -- and reset after ending a file.

If it meets a @path, it reads the file as UTF-8 (reports if not possible), splits on newlines not preceded by an odd number of \s, throws away lines starting with #, halves the number of \s before nested line terminators (round down), and iterates the remaining lines. If it sees a -- line, it disables interpretation of the rest of the file (which means just emitting the rest of the lines verbatim, only removing leading \s). If it sees a @-line when interpretation is enabled, it recursively reads and iterates that file (or reports that it can't). If a line starts with \, not immediatelyu followed by a newline, the \ is removed (warn if not followed by \, @ or #). After processing each line, the nested iterator exits, and it goes back to the outer iteration, which resets the -- flag.)

That should be doable. It is a "new format", though.
Simply expanding eagerly means inserting --s from files into a list that doesn't know about files, and then the only safe protocol is to not allow --s in files (unless the file can only be the last argument).

A quick search for "arg[s][-]file" and "@-file" found:

• Java argument files: @path syntax. ASCII or "ASCII-friendly" (like UTF-8). Separates on all whitespace, not just and line terminators, but has "..." quotes to contain spaces. # comments, \-character escapes and @-as-leading-char as verbatim escape (so no recursive files).
• GCC. Very similar to Java: Whitespace separated, with quotes to contain spaces, \ escapes, but can recursively contain other @files.
• https://www.jalview.org/help/html/features/clarguments-argfiles.html: Uses --argfile=path. Allows read multiple arg-files, and can read them recursively. Has # comments, no escaping. Doesn't allow most arguments on the command line if using an arg-file.

Other possible formats that we could adopt instead include Yaml (please don't, it just means adding - in front of each line, and sometimes needing to quote), CSV (which variant? And are the arguments then tab separated?), or a GCC/Java-like format (which there presumably are existing parsers for, but likely not in Dart).

[dcharkes]

I think adding it to package:args would be unfortunate. It would prevent us from passing any VM flags (e.g. --enable-asserts) to Dart in the @path/to/file.

[bkonyi]

I think adding it to package:args would be unfortunate. It would prevent us from passing any VM flags (e.g. --enable-asserts) to Dart in the @path/to/file.

The main upside would be that we don't need to implement this in the VM. Our VM options parsing logic is fairly complex already and could probably do with a rewrite to simplify it, so I'd be a bit worried about tacking this on to what's already there.

My other concern is that, in order to support this on the VM side, we'd have to start modifying argument lists passed to Dart programs. AFAIK, this isn't something we've done before and I'd really like to know if there's any negative implications in doing so before we go down that path.

[DanTup]

@lrhn these are all great questions, and it makes me wonder whether publishing this in pkg:args as a user-facing feature (rather than just a convention supported by Dart tools) is the right choice (at least in the first instance). For each of those questions, I can imagine some users would prefer opposite answers and if right now Dart/Flutter are the only users, does publishing it publicly just tie our hands for reconsidering any of those decisions in the future?

My main motivation here is to fix the dart test/flutter test issue so I don't have any real preferences beyond having some solution (and the files being platform/shell-agnostic and easy to write from non-Dart scripts 🙂). My original thought was that this feature should just replace the lines from the file as literal args with no escaping, nesting, etc. because it's both simple to implement and very simple to understand. The more features that are supported from the list above, the more difficult it is to understand (and what implications there might be if user input may be part of these args).

eg, I was thinking of something simple like this (this supports @file anywhere, although I think there's a reasonable argument restricting it to the last position):

Iterable<String> expandArgumentFiles(Iterable<String> args,
    // default fileReader for dart:io version: (filePath) => File(filePath).readAsStringSync()
    {ArgsFileReader? fileReader}) {
  Iterable<String> readArgumentFile(String arg) {
    try {
      final filePath = arg.substring(1);
      return fileReader != null
          ? LineSplitter.split(fileReader(filePath))
          : [arg];
    } catch (e) {
      throw ArgParserException("Failed to read arguments from file '$arg': $e");
    }
  }

  return args
      .expand((arg) => arg.startsWith('@') ? readArgumentFile(arg) : [arg]);
}

typedef ArgsFileReader = String Function(String filePath);

Some of your questions are still relevant there though (although I think many of them might be covered above... IMO either line ending, no comments, paths not URIs, relative to cwd at the start of the process, blank lines are empty strings in args - although probably we'd need to exclude trailing newlines).

[jakemac53]

This feature is really only intended for automated uses by tools invoking other tools with long command lines.

I think its totally fine (and actually best) to just ship the simplest possible feature here, which is exactly the one already proposed.

• no recursion
• it just steals the preceding @ syntax
• vm args aren't supported
• arguments with multiple lines aren't supported
• has to be the final argument (this is what we proposed right @DanTup ?)
• does no processing at all of previous arguments (so yes it can come after options/flags, even immediately following them)
• no environment variable support

Shipping this in package:args gives us flexibility to change it if we really need to in the future, and it can ship as a separate top level function in a separate library to work around the dart:io issue. Most likely, very few people will actually use it.

If people file issues asking for additional support we can address it at that time (but I strongly doubt they will).

[sigurdm]

has to be the final argument

Is that the final argument before -- or how does it interact with -- in general?

If people file issues asking for additional support we can address it at that time

Introducing e.g. escaping later will be a breaking change. Perhaps better to invest a little bit here...

[jakemac53]

Is that the final argument before -- or how does it interact with -- in general?

It doesn't. It has to be the final argument. The implementation should only look at the final argument and completely ignore the rest. The intended use case here is pretty much always just passing a single @<file> argument.

Introducing e.g. escaping later will be a breaking change. Perhaps better to invest a little bit here...

This requires a much more sophisticated implementation, testing, etc. I really do not think it is worth while.

If this is shipped as just a top level function, we can always add an argument to handle escapes later on.

[lrhn]

@jakemac53 So your proposal is:

• If the command line argument list is not empty, and it ends with a string which starts with @
• Try reading .substring(1) of that as a (UTF-8?) file, relative to CWD.
  • If successful, split the content on ASCII Unicode newline grapheme clusters.
    • (No recursion, no interpretation, no way to later recognize whether an argument came from a file?)
    • append the strings from that to the input array.
  • If not succeessful ... keep the @... argument or remove it. Report an error or not?
• Continue as normal with the (possibly new) list of arguments.

That can work.

I worry about paths in a file that depends on CWD. Maybe just recommend that all paths in a file are absolute.

I worry about not having any way to avoid this behavior, no way to escape it, if you want to have an actual last argument that starts with @. (dart add-git-user @lrhn).
One options is that if the file doesn't exist, you retain the original argument. That's very error prone, if it works most of the time, but fails disasterously if you happen to hit a name that exists, like con on Windows.
I guess, if all else fails, you can always add a -- as last argument, unless you had one of those earlier too.
(And allowing/requiring the @file to be the last element before -- will remove that option.)

I think I'd prefer making it a flag instead of special syntax: --args-file=args.txt. Then we are not stepping on any currently available syntax, we are just adding a new flag among all the other flags.
I doesn't have to be last, but that avoids having to define what -- means if it occurrs in the file. Or maybe require/define that this flag is the last flag on the line, it works just like -- in disabling any further flags already on the line. The newly read args work until they hit a --. Still no recusion, that's just an error.

[dcharkes]

I worry about paths in a file that depends on CWD. Maybe just recommend that all paths in a file are absolute.

That would be somewhat unfortunate for people typing things on the command-line. (IMHO this is a great argument why this should be implemented before it reaches main, then the Dart embedder who launches the Dart main is responsible for resolving relative paths. This is exactly the reason we have the native assets feature for dylibs. Anyway, I digress, the consensus seems to be that people want a simple, non-general solution.)

I think I'd prefer making it a flag instead of special syntax: --args-file=args.txt.

If we're building the solution in a package, I agree with this. It'd be a feature of the arg parser, not a feature of the dart binary.

[jakemac53]

There should be no distinction at all between arguments provided via file versus command line, so there is no question about how to treat paths or anything else (this is entirely up to the thing receiving the args, which gets them exactly as if they had been provided as is on the command line).

The point of this functionality is for a tool which is invoking another tool with a ton of arguments, to be able to choose to jam those arguments into a file if it thinks it might be too many to otherwise work. That is the only intended use case, and it is my opinion is that we should handle it via the simplest possible means.

[lrhn]

I want it as a flag, like --args-file, even if we use it only in the dart CLI command.
People use that command to run programs that take arguments, like dart run myscript.dart scriptArg1 scriptArg2. If a script wants to take @con as a script argument, I don't want dart to get in the way.

If they want to pass --args-file as a script arg, they'll have to use -- myscript.dart --args-file=banana, just as for any other common flag.

Adding new syntax also means having to add new ways to escape the new syntax. Using the existing flag syntax means we already have that escape.

here should be no distinction at all between arguments provided via file versus command line,

If there are no recursive arg-files, then that's fine. Otherwise the path in the recursive arg file must be handled by the arg-file expander, which means interpreting it before passing it to the real tool.
So probably don't allow recursive arg-files - just don't interpret the file.

(The recommendation to only put absolute paths into an arg file is still good, and works no matter how it interprets paths. If you write a command line, you are running from the current position. If you write a file, then you risk that file being reused. If the feature exists, then it can even be used deliberately to reuse a bunch of arguments. If the file contains only absolute paths, it'll work either way.)

If the only goal is to pass a lot of arguments, then I'd rather read them from stdin than have to create a file, and find a place to put that file (in /tmp most likely). Only works for code that doesn't use stdin for anything else, but I don't think dart does.

[jakemac53]

I'd rather read them from stdin than have to create a file

This is much more complicated to do in a general purpose way (when do you stop reading them, you still need a flag to put you in this mode, have to handle forwarding off stdin at some point to the actual tool being invoked, etc) and also not compatible with how bazel works. I don't see the advantage. The use case here is specifically for tools which are passing large numbers of arguments to other tools. Creating a file is trivial for tools (and yes, it should be under tmp/, but that's up to the tool).