Recommendation: Clarify array escaping rules in Cap’n Web spec

[brianathere]

Issue

The current specification for “literal arrays” leaves room for multiple interpretations. It is not defined whether arrays nested inside objects (or deeper structures) must be escaped, or if escaping applies only to top-level arrays. This ambiguity can lead to inconsistent behavior across implementations.

Recommendation

Update the specification to state that all arrays, regardless of context, must be escaped by wrapping them in an outer array.

Examples:

• [1,2,3] → [[1,2,3]]
• [] → [[]]
• {"items": [1,2]} → {"items": [[1,2]]}
• {"data": {"items": []}} → {"data": {"items": [[]]}}

This approach provides a single, unambiguous rule, aligns with the TypeScript reference implementation, and ensures consistent behavior across languages.

Context

While developing the Rust implementation (capn-rs), my initial serializer generated unescaped arrays inside objects. To test this approach, I patched the TypeScript reference implementation to accept them. Although workable, this added complexity and diverged from the reference behavior. After exploring both paths, I found that escaping arrays at all levels is simpler, consistent, and a better fit for interoperability.

[kentonv]

Agreed the protocol spec could be a lot more precise.

FWIW, it is definitely necessary that arrays be escaped at every level. This is because special types can appear at every level, and are always represented as an array with a type name as the first member.

E.g.:

"posts": [[
  {"id": 1, "timestamp": ["date", 1757214689123]}
  {"id": 2, "timestamp": ["date", 1758105628389]},
  {"id": 3, "timestamp": ["date", 1759131523628]},
  {"id": 4, "timestamp": ["date", 1759241345775]},
]]

Here we have an array of objects, each of which has a property "timestamp" which is a Date. Date is a special type, so is represented as an array. If nested arrays aren't escaped then this representation doesn't work.

[Dr-Emann]

I think the reason is deeper than just "arrays are escaped at every level", it's really just: expressions describing containers contain expressions.

An expression representing an object is a mapping from string keys to expressions (so if you want to express an object with an array value, the way you make an expression that represents an array is to use [[...]])

And similarly, an expression representing an array is a list of expressions as well. And this is similarly recursive, so e.g.

{
  "key": [[
    "abc",
    ["date", 1757214689123],
    [[0]]
  ]]
}

Is an expression representing an object. The expression representing the value of the "key" field of the object is an array expression. The 0th item in the array expression is an expression for a string "abc", the 1th item in the array expression is an expression for a date object, the 2th item in the array expression is another array expression which contains an integer expression.

[Dr-Emann]

I do agree on the need to clarify the protocol docs though, they talk about "literal arrays", which aren't a thing in capnweb, precisely because array expressions always contain expressions.

(Could have a ["literal", $literal_json_not_an_expression] if that were desired)

[Dr-Emann]

I worry quite a bit about an implementation of capnweb which doesn't understand that concept though.