Consider renaming `@with` for ScopedValues

[pdeffebach]

PR #50958 introduces the @with macro along with dynamic scoping generally.

DataFramesMeta.jl, the data analysis package I maintain, currently exports our own @with and will have to re-name the macro before the Julia's v1.11 release.

@with is a very valuable name. It's short and clear. Considering this, having it's use-case for ScopedValues only seems unnecessarily niche to me.

Would it be possible to re-name the with and @with to something that doesn't use such a short name?

I'm not opposed to Julia Base taking over with and @with in the abstract. But this seems like such a unique use-case that the costs don't outweigh the benefits.

[jariji]

There are quite a few macro withs on GitHub.

[vchuravy]

We do expect this to be widely used across the ecosystem once people can rely on it.

It has also been started to be used through the compatibility package ScopedValues.jl so changing it does have a non-zero cost attached to it.

[nalimilan]

I also find with quite general, with many other possible meanings. For example, withenv could have been called with too, but it's good a more specific name was used.

[Seelengrab]

Saying that a new package whose explicit purpose is to provide backwards compatibility for an as-of-yet unreleased feature of Julia should trump usages of existing packages feels a bit odd. Why shouldn't the uses of existing packages (who already have a userbase! One likely wider than ScopedValues.jl) trump Base instead? Them having to change their code and potentially slate a breaking release has a "non-zero cost" attached to it as well, and I'd wager that cost is larger than that of ScopedValues.jl.

[jariji]

I think the rule is that if you're using a name and you want a stability guarantee you need to import or qualify the name. It can be annoying for using P users, but packages getting to squeeze Base out of all the good names would be annoying too.

Anyway discussing the general policy of name priority should probably happen on Discourse rather than in the @with issue.

[pdeffebach]

I think with is not the end of the world. However I want to emphasize

We do expect this to be widely used across the ecosystem once people can rely on it.

I really can't imagine that many users will be using the new with. It might gain reasonable adoption among developers, but it just seems to me unlikely that mass of people justifies taking over such a niche name. I hope next time when discussing naming the developers spend more time thinking about this trade-off.

[CameronBieganek]

I think it is important that Base not choose an overly generic name for a specific operation. For example, to get the step size of a range, Base uses the name step. But step is a very generic name---Base should have used the name stepsize. The same principle probably applies to @with.

[CameronBieganek]

If package A depends on package B, then to avoid breakage when B adds new functions, package A should use the following pattern:

module A
using B: foo, bar
# ...
end

But is that even possible with Base? To my understanding, there is an implicit using Base when the REPL is started, so every exported name in Base is brought into scope. If that's the case, then the above pattern cannot protect a package from breakage when Base adds new names. Thus, Base should be cautious about adding very generic names like @with, especially if the name is more generic than the actual operation.

This general problem is a little depressing, because it means that if a package developer wants to follow semantic versioning, then, strictly speaking, they need to upper bound the version of Base julia, because any minor release of Base could break their package.

[jariji]

Yes it works fine in general:

julia> using ThreadsX: map

julia> parentmodule(map)
ThreadsX

But this issue is for @with.

[CameronBieganek]

I think you missed the point of my comment. The point is that packages cannot protect themselves from breakage when Base introduces new names. Thus Base should be more cautious about adding new names, especially overly generic ones. And @with seems overly generic for this use case.

[mbauman]

I think @jariji's point is that it is theoretically possible to avoid the issue — even with new Base exports — but only if you and all your users happened to follow the proposed guidance in #42080. But of course, that's not merged.

Concretely, I count 14 registered packages that export with or @with (not counting Reexports): https://juliahub.com/ui/Search?q=with&type=symbols&u=define

• https://github.com/JuliaPlots/Plots.jl
• https://github.com/JuliaData/DataFramesMeta.jl
• https://github.com/MikeInnes/Lazy.jl
• https://github.com/xukai92/WeightsAndBiasLogger.jl
• https://github.com/cscherrer/NestedTuples.jl
• https://github.com/cscherrer/TupleVectors.jl
• https://github.com/MasonProtter/StaticModules.jl
• https://github.com/complyue/ResourceManagers.jl
• https://github.com/tom-metherell/Mice.jl
• https://github.com/robertsmit/SQLCompose.jl
• https://github.com/JuliaPreludes/ContextManagers.jl
• https://github.com/aptmcl/KhepriBase.jl
• https://github.com/JuliaData/JuliaDBMeta.jl
• https://github.com/joshday/PropertyUtils.jl

[DilumAluthge]

The point is that packages cannot protect themselves from breakage when Base introduces new names.

I don't think this is true.

In a fresh REPL session:

AdminisatorsMBP:Downloads dilum$ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.0-DEV.1102 (2023-12-14)
 _/ |\__'_|_|_|\__'_|  |  Commit e9b0fa11c99 (0 days old master)
|__/                   |

julia> using DataFramesMeta: @with

help?> @with
  @with(d, expr)

  @with allows DataFrame columns keys to be referenced as symbols.

In another fresh REPL session:

AdminisatorsMBP:Downloads dilum$ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.0-DEV.1102 (2023-12-14)
 _/ |\__'_|_|_|\__'_|  |  Commit e9b0fa11c99 (0 days old master)
|__/                   |

julia> module Foo
       using DataFramesMeta: @with
       end
Main.Foo

help?> Foo.@with
  │ Warning
  │
  │  The following bindings may be internal; they may change or be removed in
  │  future versions:
  │
  │    •  Main.Foo.@with

  @with(d, expr)

  @with allows DataFrame columns keys to be referenced as symbols.

---

Pkg status:

julia> import Pkg; Pkg.status()
Status `~/.julia/environments/v1.11/Project.toml`
  [1313f7d8] DataFramesMeta v0.14.1

[CameronBieganek]

In a fresh REPL session:

julia> using DataFramesMeta: @with

That's not the package protecting itself from breakage, that's the package user protecting themselves from breakage. The current status quo is that using DataFramesMeta is allowed (even encouraged by the Julia manual). And if you have a compatible set of packages installed by the package manager, e.g. DataFramesMeta version 0.14.1 on Julia version 1.11, then the following should work, because supposedly the package manager installed a compatible set of packages:

using DataFrames, DataFramesMeta
df = DataFrame(x = 1:2);
@with(df, :x .+ 1)

Anyhow, I opened up an issue for the more general issue of package breakage from new exported names in minor releases of Base Julia: #52540.

[pdeffebach]

DataFramesMeta.jl is meant to be a REPL-package for data cleaning and data exploration, so even if Julia recommends against using for packages, the @with export would still be a problem in this case.