feat: StreamableShell with exec_command and write_stdin tools

[bolinfest]

This introduces a complementary set of tools, exec_command and write_stdin, which are designed to facilitate working with long-running processes in a token-efficient manner.

To test:

codex-rs$ just codex --cd .. --config experimental_use_exec_command_tool=true

Though the above alone is unlikely to convince the model to use the new tools because of this bit in the base instructions:

codex/codex-rs/core/prompt.md

Lines 266 to 271 in e4c275d

	## Shell commands
	When using the shell, you must adhere to the following guidelines:
	- When searching for text or files, prefer using `rg` or `rg --files` respectively because `rg` is much faster than alternatives like `grep`. (If the `rg` command is not found, then use alternatives.)
	- Read files in chunks with a max chunk size of 250 lines. Do not use python scripts to attempt to output larger chunks of a file. Command line output will be truncated after 10 kilobytes or 256 lines of output, regardless of the command used.

So you probably need to add:

--config experimental_instructions_file=/some/other_instructions.md

where /some/other_instructions.md suggests using exec_command and write_stdin instead of shell.

[hanson-openai]

looks like it was defined as an int elsewhere (also we don't output a session ID from the exec outputs so the model hallcuinates one)

{"type":"function_call_output","call_id":"call_ZH0rYAapH2v1P5XNHOTGO7Fk","output":"failed to parse function arguments: invalid type: string \"e6d45cfc-81f2-44d6-b52b-6df1f8c3b9d0\", expected u32 at line 1 column 68"}

[hanson-openai]

will probably work best if the output is either in the form:

Wall time: {:.3g} seconds
Process exited with code {code}
# or, if running: Process running with session ID {session_id}
Warning: truncated output (original token count: {tokens})  # (if truncated)
Output:
{actual text output}

OR if it's more convenient to use json:

{
  "wall_time": round(wall_time, 3),
  "exit_code": code,
  "session_id": session_id, # should not be set if exit_code is set
  "original_token_count": count, # only if truncated
  # NOTE: you'll want to dump the JSON w/o whitespace to reduce token usage
  "output": {"1":line1,"2":line2,...}
}

Both exec_command and write_stdin can return the exact same shape of output

[hanson-openai]

FYI: the semantics of the internal implementation are actually to collect all output within yield_time_ms (uncapped) and then truncate the middle tokens/characters after receiving all the output.

Otherwise the agent has no real way of "flushing out" outputs if a command has a ton of logspew

[hanson-openai]

reference description:

Write characters to an exec session's stdin. Returns all stdout+stderr received within yield_time_ms.
Can write control characters (\u0003 for Ctrl-C), or an empty string to just poll stdout+stderr.

[hanson-openai]

In the Python version I found that read() may throw a BlockingIOError sometimes, in which case i would wait a bit and try again https://docs.python.org/3/library/exceptions.html#BlockingIOError

not sure if that's a idiosyncracy from the python path, or is it handled internally by the rust standard libs somewhere?

[hanson-openai]

nice

[hanson-openai]

would be good to put some message in the middle so it's more visible where the truncation happened, e.g.

…{truncated} tokens truncated…

[bolinfest]

@hanson-openai I think this is the read that you are worried about BlockingIO. I'll add some logic.

[bolinfest]

@hanson-openai addressed your feedback and updated tests, but I'll also ask someone from Codex to take a look

[dylan-hurd-oai]

Approving to unblock, will review more carefully tomorrow.

My main current thought is that an integration test would be really helpful here!

[bolinfest]

Did you miss the integration test in session_manager.rs?