loading: More extension loading cleanup (#60146)

In #59863, I removed a special case that allowed extensions to be
identified cross-environment. As it turns out, `locate_package` has a
similar special case, except this one is even weirder (because it will
definitely break if any package other than the intended base package is
found).

However, it turns out that removing this special case breaks stdlib
precompilation. Why you ask? Well, when we precompile stdlibs we set the
LOAD_PATH to `@stdlib:$(pwd)/stdlib` and then run the precompile
function (which calls locate_package directly). Note that `@stdlib` is
an implicit env that we symlink together, while `$(pwd)/stdlib` is an
explicit env with Project and Manifest. However, these projects and
manifests are somewhat malformed, because we don't actually have the
stdlibs in our depot, so they cannot be used for loading.

What's happening here is that it turns out we had a separate bug where
we cannot properly load extensions from implicit environments unless the
parent package has previously been loaded. This is usually the case
(because extensions by definitions have their parent packages loaded),
but is not the case during an explicit precompile.

The tangled setup previously described found `Statistics` in the first
environment, then found Statistics' SparseArrayExt in the second
environemnt, which then computed the extension path by looking up
`Statistics` in the first environment again.

With loading properly fixed to find extensions in implicit environments
(which is painful since we need to scan each project.toml in each
directory), everything goes through and we find all the packages and the
appropriate extensions in the implicit environment.

I don't think any of this was ever intentionally designed this way but
just happened to be a combination of bugs covering for each other.

I also don't think that this fallback case will actually trigger in
practice although it does of course trigger when looking up any package
that is not actually present in the environment.

As such, it is advisable to avoid large implicit environments at the
start of the loadpath. `@stdlib` is last in the default setup and our
other environments are explicit, so again I don't forsee this to happen
too often in practice, but we might want to consider making `@stdlib` an
explicit environemnt also to explicitly record everything.

Author: Keno

base/loading.jl

@@ -824,7 +824,10 @@ function manifest_uuid_path(env::String, pkg::PkgId)::Union{Nothing,String,Missi
         # if env names a directory, search it
         proj = implicit_manifest_uuid_path(env, pkg)
         proj === nothing || return proj
-        # if not found
+        # if not found, this might be an extension - first we fast path needing
+        # to scan the whole directory for a matching extension by peeking at
+        # EXT_PRIMED. However, this only works if the parent package was loaded.
+        # This is usually the case, but not always, e.g. in precompilation.
         triggers = get(EXT_PRIMED, pkg, nothing)
         if triggers !== nothing
             parentid = triggers[1]
@@ -836,6 +839,10 @@ function manifest_uuid_path(env::String, pkg::PkgId)::Union{Nothing,String,Missi
                     mby_ext === nothing || return mby_ext
                 end
             end
+        else
+            # We still need to scan the whole directory for extensions.
+            ext_path, ext_proj = implicit_env_project_file_extension(env, pkg)
+            ext_path === nothing || return ext_path
         end
     end
     return nothing
@@ -1130,8 +1137,8 @@ function explicit_manifest_uuid_path(project_file::String, pkg::PkgId)::Union{No
             uuid = get(entry, "uuid", nothing)::Union{Nothing, String}
             extensions = get(entry, "extensions", nothing)::Union{Nothing, Dict{String, Any}}
             if extensions !== nothing && haskey(extensions, pkg.name) && uuid !== nothing && uuid5(UUID(uuid), pkg.name) == pkg.uuid
-                parent_path = locate_package(PkgId(UUID(uuid), name))
-                if parent_path === nothing
+                parent_path = explicit_manifest_entry_path(manifest_file, PkgId(UUID(uuid), name), entry)
+                if parent_path === nothing || parent_path === missing
                     error("failed to find source of parent package: \"$name\"")
                 end
                 p = normpath(dirname(parent_path), "..")