proposal: spec: typed struct tags

[Merovius]

Proposal: Typed struct tags

This is a fully fleshed out version of a design I sketched on #23637. It is prompted by discussion in #71664.

I propose to expand the definition of struct tags to allow a list of constant expressions, in addition to the existing string tag. These typed tags must be a comma-separated list enclosed in parenthesis. Packages can then define types and constants that can be used as struct tags to customize behavior. To demonstrate the syntax, here is how encoding/json could take advantage of this facility (see below for the definitions of these tags):

type Before struct {
    F1 T1        `json:"f1"`
    F2 T2        `json:"f2,omitempty"`
    F3 T3        `json:",omitzero"`
    F4 T4        `json:"f4,case:ignore"`
    F5 time.Time `json:",format:RFC3339"`
    F6 time.Time `json:",format:'2006-01-02'"`
    F7 T7        `json:"-"`
    F8 T8        `json:"-,"`
}

type After struct {
    F1 T1        {json.Name("f1")}
    F2 T2        {json.Name("f2"), json.OmitEmpty}
    F3 T3        {json.OmitZero}
    F4 T4        {json.Name("f4"), json.IgnoreCase}
    F5 time.Time {json.Format(time.RFC3339)}
    F6 time.Time {json.Format("2006-01-02")}
    F7 T7        {json.Ignore}
    F8 T8        {json.Name("-")}
}

type Mixed struct {
    F1 T1
    F2 T2        `yaml:"f2,omitempty"`
    F3 T3                              {json.OmitZero}
    F4 T4        `yaml:"f4"`           {json.Name("f4"), json.IgnoreCase}
}

The rest of the proposal describes the changes needed to the language, the reflect and go/ast packages and as an example of use, how the encoding/json/v2 API can take advantage of them.

The proposal is fully backwards compatible, so it could simply be enabled if a module uses Go 1.N, without requiring any additional migration.

Rationale

Struct tags are currently opaque strings, as far as the language concerned.

The reflect package defines a conventional mini-language for them as key-value pairs with values being quoted. Packages then further define micro-languages for those values. For example, encoding/json/v2 defines them as a comma-separated list of options. Some of these options, in turn, are specified in their own nano-languages. For example:

The "format" option specifies a format flag used to specialize the formatting of the field value. The option is a key-value pair specified as "format:value" where the value must be either a literal consisting of letters and numbers (e.g., "format:RFC3339") or a single-quoted string literal (e.g., "format:'2006-01-02'"). The interpretation of the format flag is determined by the struct field type.

The last sentence hints at the fact that struct field types might then define pico-languages for formats.

With all these layers of bespoke syntax, each with its own rules for quoting and set of allowed and disallowed characters, it becomes increasingly easy to make mistakes. A common error, for example, is to omit quotes from struct tags and write e.g. "json:foo".

Given that struct tags, as far as the language is concerned, are simply opaque strings the compiler rightfully does not complain about this and runs the program, leaving the developer to figure out why JSON marshaling does not work.

Linters can help, but not every third-party package using struct tags ships a linter. And even if they do, they might not commonly be installed or run.

All of this syntax also needs to be parsed at runtime, which requires extra code and potentially introduces overhead.

Lastly, the outermost key of struct tags is not namespaced. The json key is simply an arbitrary prefix. This is not a problem for struct tags that are sufficiently well-known (e.g. those used by the standard library), but with third party packages clashes become increasingly likely. For example, there are multiple YAML parsing packages using yaml: struct tags with their own bespoke syntax.

This proposal solves these problems by replacing the opaque string with constants of arbitrary types. If a name is mistyped, the syntax is erroneous or an invalid value is used, the compiler directly complains. And as types are already namespaced to packages (which are unique within the program) there can be no clashes.

Language changes

The change to the language consists of this diff to the Struct types section:

 <pre class="ebnf">
 StructType    = "struct" "{" { FieldDecl ";" } "}" .
-FieldDecl     = (IdentifierList Type | EmbeddedField) [ Tag ] .
+FieldDecl     = (IdentifierList Type | EmbeddedField) [ Tag ] [ TypedTags ] .
 EmbeddedField = [ "*" ] TypeName [ TypeArgs ] .
 Tag           = string_lit .
+TypedTags     = '{' ExpressionList '}' .
 </pre>
… 
 <p>
-A field declaration may be followed by an optional string literal <i>tag</i>,
-which becomes an attribute for all the fields in the corresponding
-field declaration. An empty tag string is equivalent to an absent tag.
-The tags are made visible through a <a href="/pkg/reflect/#StructTag">reflection interface</a>
-and take part in <a href="#Type_identity">type identity</a> for structs
-but are otherwise ignored.
+A field declaration may be followed by an optional string literal <i>tag</i>,
+as well as an optional list of <i>typed tags</i>. Both become attributes for
+all the fields in the corresponding field declaration. Typed tags must be <a
+href="#constant_expressions">typed constant expressions</a> and their types
+must not be <a href="#Predeclared_identifiers">predeclared types</a>. An
+absent string tag is equivalent to an empty string. Tags are made visible
+through a <a href="/pkg/reflect/#StructField">reflection interface</a> and take
+part in <a href="#Type_identity">type identity</a> for structs but are
+otherwise ignored.
 </p>

Further, in the "Type identity" section, we might want to add "[…] and identical tags in the same order" for clarity.

The intention is to allow a single string tag for backwards compatibility and migration, which is interpreted according to the current semantics. All other tags must be user-defined types. As using predeclared types might lead to ambiguous interpretations (see above about the namespacing issue), it doesn't seem like a steep cost to mostly rule them out. Should we find a good reason to do so, we can remove this restriction.

Changes to reflect

Access to typed tags is given via new reflect APIs:

type StructField struct {
    // …
    Tag  StructTag  // field tag string
    tags structTags // other field tag constants
    // …
}

// Tags returns an iterator over the tag constants of f.
func (f StructField) Tags() iter.Seq[Value]

// SetStructTags overwrites the field tag constants of f, for use with
// [StructOf].
// 
// All tags must have user-defined string, boolean or numeric types.
func (f *StructField) SetTags(tags ...Value)

// StructTagsFor returns an iterator over all tags of type T.
func StructTagsFor[T any](StructField) iter.Seq[T]

We can not simply make the field an exported slice, because that would make it possible to modify its backing array. So we would have to ensure that for any StructField we return, the backing array is not shared with the internal representation, which requires an allocation.

For similar reasons, Values yielded by the iterator are not addressable.

The type of tags can not be a slice either. StructField is currently a comparable type and a slice field - even an unexported one - would change that. That would break compatibility. So we must find a representation that is comparable with the right semantics to preserve type-identity (that is, two StructFields should be identical, if they contain the same tags in the same order). One such representation is a pointer to a singleton, de-duplicated with a custom map that can work with slice-keys. Another possibility would be to encode them into a string.

In practice, StructTagsFor is the primary way users should interact with this API. They would write, for example:

for i := range structType.NumField() {
    f := structType.Field(i)
    for t := range reflect.StructTagsFor[MyFlag](f) {
        switch t {
        case FlagFoo:
            // do something fooy
        case FlagBar:
            // do something bary
        }
    }
    name, ok := xiter.First(reflect.StructTagsFor[MyName](f))
    if ok {
        // there's at least one MyName tag with value name.
    }
}

The API (in particular because tags are stored in an unexported field) allows reflect to cache a list of tags for a given type/field combination. That is likely unnecessary in practice, but it at least keeps the possibility open for StructTagsFor to be more efficient than simply filtering StructField.Tags().

Some parts of the reflect code likely must be modified to handle type identity correctly.

Changes to go/ast

We expose the tags as an extra struct field:

type Field struct {
    // …
    Tag  *BasicLit // field string tag; or nil
    Tags []Expr    // field tag constants
    // …
}

Other go/* packages likely must be modified as well, to implement type-identity correctly and format the new syntax.

Exemplary changes to encoding/json

While not part of this proposal, it is instructive to consider how it can be used with the example of encoding/json/v2. We could add this new API to the json package:

// Name is a struct tag type to specify the JSON object name override for the Go
// struct field. If the name is not specified, then the Go struct field name is
// used as the JSON object name. By default, unmarshaling uses case-sensitive
// matching to identify the Go struct field associated with a JSON object name.
type Name string

// Flags is a struct tag type to customize JSON parsing behavior.
type Flags int

const (
    // Ignore specifies that a struct field should be ignored with regard to
    // its JSON representation.
    Ignore Flags = iota
    // OmitZero specifies that the struct field should be omitted when
    // marshaling, if (etc…)
    OmitZero
    // OmitEmpty specifies that the struct field should be omitted when
    // marshaling, if (etc…)
    OmitEmpty
    // String specifies that [StringifyNumbers] be set when marshaling or
    // unmarshaling (etc…)
    String
    // Inline specifies that the JSON representable content of this field type
    // is to be promoted as if they were specified in the parent struct. (etc…)
    Inline
    // Unknown is a specialized variant of the inlined fallback (etc…)
    Unknown
)

// Case is a struct tag type to specify how JSON object names are matched with
// the JSON name for Go struct fields, when unmarshaling.
type Case int
 
const (
    // IgnoreCase specifies that name matching is case-insensitive where dashes
    // and underscores are also ignored. If multiple fields match, the first
    // declared field in breadth-first order takes precedence.
    IgnoreCase Case = iota
    // StrictCase specifies that name matching is case-sensitive. This takes
    // precedence over the [MatchCaseInsensitiveNames] option.
    StrictCase
)

// Format is a struct tag type to specify a format flag used to specialize the
// formatting of the field value. The interpretation of the format flag is
// determined by the struct field type. 
type Format string

This includes doc strings, for comparison with the existing documentation. They are largely copied over, but notice that a bunch of prose related to escaping and other formatting is omitted.

An example of how this would look when migrating string tags to the new API:

type Before struct {
    F1 T1        `json:"f1"`
    F2 T2        `json:"f2,omitempty"`
    F3 T3        `json:",omitzero"`
    F4 T4        `json:"f4,case:ignore"`
    F5 time.Time `json:",format:RFC3339"`
    F6 time.Time `json:",format:'2006-01-02'"
    F7 T7        `json:"-"`
    F8 T8        `json:"-,"`
}

type After struct {
    F1 T1        {json.Name("f1")}
    F2 T2        {json.Name("f2"), json.OmitEmpty}
    F3 T3        {json.OmitZero}
    F4 T4        {json.Name("f4"), json.IgnoreCase}
    F5 time.Time {json.Format(time.RFC3339)}
    F6 time.Time {json.Format("2006-01-02")}
    F7 T7        {json.Ignore}
    F8 T8        {json.Name("-")}
}

Note how the format string tag requires to syntactically differentiate between using a common layout and a custom one for time.Time formatting (by using single-quotes around the value), while the typed tags are similarly convenient without needing that distinction. The exception are formats that cannot be covered by a layout string, such as unix, sec or nano. However, these can be special-cased.

In #71664 we discussed giving access to struct fields to UnmarshalJSONFrom/MarshalJSONTo. With this proposal, that API could look like this:

// FieldTagsFor returns an iterator over the field tags with a given type, set
// on the currently parsed field.
func FieldTagsFor[T any](Options) iter.Seq[T]

Discussion

Composite types

This proposal intentionally leaves out the possibility of using composite types like structs or slices as tags.

Go currently does not have a notion of constants for composite types. As we should preserve the property of struct tags to be statically analyzable, we would want them to be constants. So to support composite tags, we would need to introduce some notion of struct-constant, which seems overkill for a small language feature like this. However, should Go ever gain general support for composite constants, they would slot seamlessly into this proposal.

In the meantime the JSON example should illustrate that it is possible to encode quite complex options into constants as well.

Repeated tags

The proposal allows to use multiple tags of the same type, including repeating the same tag value. It would be possible to prevent that, by requiring that there must be at most one tag per type.

One advantage of that would be that it simplifies the reflect API to no longer require iterators, when looking up a single type:

func StructTagFor[T any](StructField) (tag T, found bool)

It would possibly catch mistakes of specifying mutually exclusive tags. On the other hand, there might be cases that could assign meaning to having the same tag type used multiple times (to effectively emulate slice tags). It is unclear whether these add up to an advantage or a disadvantage.

One downside is the modeling of flag-like tags (like json.OmitZero etc. above). As tag types can not be repeated, each of these would require its own type. Furthermore, those types could still have multiple values and it is unclear what that would mean; for example, if json.OmitZero is a boolean constant and true, what would !json.OmitZero mean?

The types could be unexported, with an exported constant, e.g.

type omitZero bool

const OmitZero = omitZero(true)

However, this would prevent third-party packages from retrieving such flags, which might be desirable e.g. for drop-in replacement JSON parsing packages.

They could also be a single type, used as a bitmask:

type Flags int

const (
    Ignore Flags = (1<<iota)
    OmitZero
    OmitEmpty
    String
    Inline
    Unknown
)

type X struct {
    F int {json.Name("f"), json.OmitEmpty|json.Empty}
}

However, the | separator between some tags and not others looks out of place.

Overall, the disadvantages seem to outweigh the advantages. And if we really feel the need to simplify the API, we can do that as a helper:

// LookupStructTagFor returns the first tag of the given type found.
func LookupStructTagFor[T any](f StructField) (tag T, found bool) {
    for tag = range StructTagFor[T](f) {
        return tag, true
    }
    return tag, false
}

Complex expressions

The proposal allows any expression for use in tags, including arithmetic expressions. In practice, tags should likely be restricted to selector-expressions (for named constants) and conversions (for "parameterized tags", like json.Name). We could restrict the syntax to those, just like they are currently restricted to string literals, not string constants.

The proposal does not do that mainly for simplicity. We could decide to add the restriction in the beginning and only expand it, if a need for more complex expressions is demonstrated over time. We need to be aware that expanding the syntax later would potentially break tools assuming the restrictions, though.

A more restricted syntax would allow us to specify a canonical ordering of tags, which could be enforced by the API and maintained by gofmt. This might help readability.

Compile time dependencies

One downside of this proposal is that typed tags introduce a compile time dependency on the package defining them. Currently, if a package contains

package a

type X struct {
	Foo string `json:"foo"`
}

then importing a does not require importing encoding/json. With this proposal, the package would have to be

package a

import "encoding/json"

type X struct {
	Foo string {json.Name("foo")}
}

which adds an import.

This means that if a program contains types which define JSON marshaling options, but which does not end up actually (un)marshaling any JSON, compile times, binary size and initialization time can go up. Similar, of course, for other packages defining tags.

Under the assumption that the types used as tags do not have methods on them (or only methods that don't call into the rest of the package - e.g. most fmt.Stringer implementations should be fine) and are only used as tags, the linker should be able to eliminate most if not all of the code from the package as never used. Only type definitions, constant values and whatever is needed to initialize the package must actually be linked in.

But some impact is unavoidable. Authors of packages which define tags should be aware of this and encouraged to avoid global variables, init functions and methods on the tag types, as much as possible.

Syntax

The syntax includes curly braces to group the typed tags. These have two functions. One is to separate the legacy string tag from the typed tags, so it is clearly defined which tag is provided via which API.

The other function is to prevent an ambiguity with embedded fields. Say the parenthesis would not be part of the grammar:

type X struct{
    A B // field A of type B, or embedded field A with struct tag B?
}

The ambiguity could still be resolved by explicitly providing the implied empty string tag with embedded fields:

type X struct{
    A B     // field A of type B
    A "", B // embedded field A with tag B
}

But this looks awkward and the compiler would have to at least suggest this, if it encounters a constant where it expects a type in a field declaration.

The choice of syntactical construct is up for discussion. Curly braces seem to work, syntactically. Square brackets where considered but do not work, as they create a syntactic ambiguity:

type X struct {
	A B [C] // Field A of generic type B, instantiated with type argument C
}

type B[T any] struct{}

Parenthesis do not work either:

type X struct {
    A (B) // Field A of type B, or embedded Field A with tag B?
}

There could also be a single token inserted between the string tag and the typed tags. We could, for example, introduce a new token @ as a nod to Python decorators:

type X struct{
    A B         // field A of type B
    A       @ B // field A with tag B
    A B     @ C // field A with type B and tag C
    A B `C`     // field A with type B and string tag `C`
    A B `C` @ D // field A with type B, string tag `C` and tag D
}

The choice of punctuation is limited by the fact that they must not be binary operators:

type X struct{
    A B | C // field A of type B with tag C, or embedded field A with tag B|C?
}

Unary operators should be okay, as neither ! nor ~ can start a type. , and ; are ruled out as they are list-separators and . is ruled out because struct{ A B . C } (could be type B.C or type B, tag C).

So, of the currently defined punctuation characters, we are left with !, ~, :=, ..., :.

We could also drop the , as a list-separator for the tags and instead use it to separate the tags.

Lastly, we could use a keyword, e.g. struct{ A B const C }.

Tools

Tools that operate on struct tags might have to change. On the other hand, at least some of them would become now obsolete, because the most likely use case of such tools is to lint the struct tag syntax.

We could provide a tool to automatically migrate tags from standard library packages. However, such a transformation would not preserve the semantics of a program, if a third party package consumes the string tag as well. So such a tool should not be run automatically.

[Merovius]

cc @griesemer (spec change), @dsnet (JSON), @adonovan (tooling)

[dsnet]

Thanks, this is very well thought out.

I agree with the decision of making this only support constants, as that avoids a major problem with #23637.

mini-language ... micro-language ... nano-language ... pico-language

😅 At least, we didn't get the level of femto-language yet.

Generally, I find the readability of what tags are present to be more readable, but the readability of the entire struct unfortunately takes a hit since the typed tags can look syntactically identical to that of a field type. String literals didn't have this problem because they could not possibly be mistaken as a field type. I do think a syntactic token (whether it be a ':' as you suggested or a surrounding [ and ] as proposed in #23637) to indicate the presence of tags to be visually helpful and avoids the embedded fields problem that you noted.

While this may be out of scope, I do think it's possibility in the future is worth considering. Today, we have struct field tags, but I've often come across the need for type tags as well. As an example, there exist Go structs, where omitzero is declared on every single field. It would be convenient to indicate that omitzero is by default applied to all fields in the Go struct by declaring this tag on the type itself. I tried writing a design for this, but ran into a syntactic ambiguity issue somewhat similar to your embedded type situation:

type Foo struct {
    ...
} `json:omitzero` // type tag that indicates that `omitzero` is declared on all fields in Foo

but becomes ambiguous when trying to apply it on inline type declaration:

type Foo struct {
    Bar struct {
    } `json:omitzero` // is this a tag for the Bar field or the inlined struct type?
}

I don't have any great solutions (which is why I never proposed type tags), but it would be nice if there was a syntactic construct that allows us to extend tags to be declared on types as well without ambiguity.

[Merovius]

I do think a syntactic token (whether it be a ':' as you suggested or a surrounding [ and ] as proposed in #23637) to indicate the presence of tags to be visually helpful and avoids the embedded fields problem that you noted.

The brackets don't work because A B [C] is a field with generic type B instantiated with C.

I tend to agree that having a syntactic indicator would be more readable, but couldn't think of one I really liked. So I just brought it up so others can continue bikeshedding. I don't super like : myself and would prefer some form of brackets, but we already have so many in Go, at this point, that ambiguities start cropping up.

[TapirLiu]

type StructField struct {
    // …
    Tag  StructTag  // field tag string
    tags structTags // other field tag constants
    // …
}

StructTag is a comparable, Will structTags be guaranteed to be? If not, I'm not sure whether this will cause some troubles.

[Merovius]

@TapirLiu

The type of tags can not be a slice either. StructField is currently a comparable type and a slice field - even an unexported one - would change that. That would break compatibility. So we must find a representation that is comparable with the right semantics to preserve type-identity (that is, two StructFields should be identical, if they contain the same tags in the same order). One such representation is a pointer to a singleton, de-duplicated with a custom map that can work with slice-keys. Another possibility would be to encode them into a string.

[jimmyfrasche]

Regardless of specifics, very much 👍 on the general concept.

I'd like the : even if it's strictly not necessary just to have something to hint the transition visually, regardless of formatting. / or | would also work, without having to introduce a new token. (@dsnet You could also use a different one for type tags which would let you specify both a type tag and a field tag in your struct-in-a-struct example.)

I think allowing const struct values when all the fields may be constant would be a sufficiently useful change to go with this, allowing many settings to be bundled up in one place and easily reused on many fields by making a single const declaration. Some relevant posts of mine in previous threads, for reference: #6386 (comment) #23637 (comment)

[jimmyfrasche]

If a delimiter is used, I think the syntax should be

Tags = string_lit | ':' ExpressionList .

or, in words, you can continue to use a single string lit by juxtaposition as before or you can you can opt in to the new way/features by starting with the delimiter. It would be safe to rewrite F T "L" to F T : "L" as they mean the same thing.

[Merovius]

@jimmyfrasche The issue I have with that is that "the new way" then has to also allow string literals and you have to define which string literal goes into the "old" Tag field. That's because (as the proposal text explains) you must be able to use both on the same field, when migrating and you have two packages, only one of which supports "the new way". So I really don't think it should be an alternative between them - it should be an optional string tag and an optional group of constant expressions.

So, I've thought about it some more and I am convinced of the benefit of having an explicit syntactical marker. But I'd put it between the string tag and the rest of the tags. That avoids the ambiguity of which string literal goes into the fallback: There is a clearly separate optional string tag and then there is a group of typed tags. The first is accessed using the old API, the last is accessed using the new API. That also removes the need for the somewhat artificial restriction to user-defined types.

Now, a plain separator between the string tag and the rest looks kind of strange. But some kind of bracket really makes sense, because it groups the… group of typed tags. Square brackets don't work, but I believe parenthesis do. I initially ruled them out, because expressions can start with (. But that doesn't matter, as long as there is a required enclosing pair of parenthesis. So, that would be e.g.

type X struct {     
    Foo             string    `yaml:"foo"` (json.Name("foo"), json.OmitZero)                   
    Bar             int       `yaml:"bar"`                      
    Baz             float64                (json.Name("baz"))                            
    StrangeButValid time.Time              ((((42+23))))              
}    

I think I've convinced myself of the advantages of this enough, that I will rewrite the proposal with this. Unless someone tells my why it categorically doesn't work. And we can still bikeshed the exact syntactic distinction - I like parenthesis, but it mostly works the same with : or @ or «…».

[nussjustin]

Now, a plain separator between the string tag and the rest looks kind of strange. But some kind of bracket really makes sense, because it groups the… group of typed tags. Square brackets don't work, but I believe parenthesis do.

Assuming parenthesis would work I'm in favor of this. I also tried a few different separators and disliked most of them. The only one I kinda liked was %, but parenthesis also make more sense to me.

[nussjustin]

The proposal allows any expression for use in tags, including arithmetic expressions. In practice, tags should likely be restricted to selector-expressions (for named constants) and conversions (for "parameterized tags", like json.Name).

Having played around a bit with this proposal, I'm in favor of adding this restriction. Even more so after seeing the StrangeButValid field @Merovius last example.

The proposal allows to use multiple tags of the same type, including repeating the same tag value. It would be possible to prevent that, by requiring that there must be at most one tag per type.

I think that tags with the same type should be allowed as long as the values are not repeated. So json.Name("a"), json.Name("b") would be ok, but not json.Name("a"), json.Name("a").

[Merovius]

I think that tags with the same type should be allowed as long as the values are not repeated. So json.Name("a"), json.Name("b") would be ok, but not json.Name("a"), json.Name("a").

Why this? If it makes sense to have the same tag multiple times, surely it makes sense to allow it to be used twice - we wouldn't restrict a slice to only have different elements, after all. Likewise, if there is a problem with having the same tag type twice, surely it is if the values are different so that it is no longer clear which value has precedence.

I think that's kind of the problem with any restriction: There are just different ways to use tag types, for which different kinds of restrictions make sense. For json.Name it makes no sense to even have more than one of the same type, especially not if they differ. On the other hand, for json.Flags it is necessary to have more than one of the same type, but they have to be able to differ.

The only reason I would think it makes sense to add a restriction on repetition is, if it allows us to simplify the API. And the problems with that are discussed in the proposal text.