WebApplicationFactory doesn't set solution relative content root if you switch to slnx solution

[kimsey0]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

WebApplicationFactory tries setting the content root based on a number of sources (a builder setting, MvcTestingAppManifest.json, and the assembly metadata) before falling back to setting a solution-relative content root. This looks specifically for a .sln file, which means any test that relies on it breaks when migrating to the new SLNX solution format.

Expected Behavior

If ASP.NET Core supports falling back to searching for a solution file and setting the content root relative to it, it should also be able to locate slnx solution files.

Steps To Reproduce

The simplest way to trigger this is with a WebApplicationFactory that uses an entry point in the test assembly, instead of in a separate web API project. Create a solution with a solution file, a project file, and a test:

SlnxReproduction.slnx

<Solution>
  <Project Path="Tests/Tests.csproj" />
</Solution>

Tests/Tests.csproj

<Project Sdk="Microsoft.NET.Sdk.Web">
<!-- Can also be Microsoft.NET.Sdk with a reference to an ASP.NET Core project. -->

  <PropertyGroup>
    <TargetFramework>net9.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.AspNetCore.Mvc.Testing" Version="9.0.3" />
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.12.0" />
    <PackageReference Include="xunit" Version="2.9.2" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.8.2" />
  </ItemGroup>

  <ItemGroup>
    <Using Include="Xunit" />
  </ItemGroup>

</Project>

Tests/SlnxTest.cs

using Microsoft.AspNetCore.Mvc.Testing;

namespace Tests;

public class SlnxTest(WebApplicationFactory<Startup> factory) : IClassFixture<WebApplicationFactory<Startup>>
{
    [Fact]
    public void CreateClient_DoesNotThrow() => factory.CreateClient();
}

public class Startup;

Then run the test with dotnet test.

Exceptions (if any)

System.InvalidOperationException : Solution root could not be located using application root C:\Project\Path\bin\Debug\net9.0\

Stack Trace:
 at Microsoft.AspNetCore.TestHost.WebHostBuilderExtensions.UseSolutionRelativeContentRoot(IWebHostBuilder builder, String solutionRelativePath, String applicationBasePath, String solutionName)
 at Microsoft.AspNetCore.TestHost.WebHostBuilderExtensions.UseSolutionRelativeContentRoot(IWebHostBuilder builder, String solutionRelativePath, String solutionName)
 at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.SetContentRoot(IWebHostBuilder builder)
 at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.<ConfigureHostBuilder>b__22_0(IWebHostBuilder webHostBuilder)
 at Microsoft.Extensions.Hosting.GenericHostWebHostBuilderExtensions.ConfigureWebHost(IHostBuilder builder, Func`3 createWebHostBuilder, Action`1 configure, Action`1 configureWebHostBuilder)
 at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.ConfigureHostBuilder(IHostBuilder hostBuilder)
 at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.EnsureServer()
 at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.CreateDefaultClient(DelegatingHandler[] handlers)
 at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.CreateDefaultClient(Uri baseAddress, DelegatingHandler[] handlers)
 at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.CreateClient(WebApplicationFactoryClientOptions options)
 at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.CreateClient()```

.NET Version

9.0.202

Anything else?

ASP.NET Core 9.0.3

[kimsey0]

This could be fixed by adding a wildcard to the .sln search pattern. See #61305. However, I don't know if this would be considered a breaking change, since you could technically have a .slnx file that then "shadows" a .sln file in a parent folder. Tests, documentation, etc. should of course be updated too.

[kimsey0]

Actually, using .sln? probably isn't a good idea, since it would also match solution filter files (.slnf).

[Aaronontheweb]

Is there a workaround for this?

[martincostello]

Doing something like this:

builder.UseSolutionRelativeContentRoot(Path.Combine("src", "MyProjectName"), "*.slnx");

[siewers]

@martincostello I tried this, but unfortunately, I couldn't get it to work. The SetContentRoot method always calls the UseSolutionRelativeRoot with the default *.sln extension if the MvcTestingManifest.json doesn't contain the assembly being tested. In this case, the test assembly itself is the content root, which is failing to be resolved.

I ended up patching the MvcTestingManifest.json file by adding the test assembly, which fixes the issue, although it's a bit of a hack. On the contrary, I'm not sure this case is how the WebApplicationFactory was intended to be used.

Regardless, since *.slnx is officially supported, the WebApplicationFactory should also support it.

This is the code I'm using now:

public static class MvcTestingAppManifestHelper
{
    private const string ManifestFileName = "MvcTestingAppManifest.json";
    private static readonly JsonSerializerOptions JsonSerializerOptions = new() { WriteIndented = true };

    public static void AddAssemblyToManifest(Assembly assembly)
    {
        if (!File.Exists(ManifestFileName))
        {
            return;
        }

        // The manifest file is a dictionary of Assembly.FullName to ContentRoot.
        var manifest = JsonSerializer.Deserialize<Dictionary<string, string>>(File.ReadAllText(ManifestFileName))!;

        // Internally, the tilde is used to translate the content root to the AppContext.BaseDirectory.
        if (manifest.TryAdd(assembly.FullName!, "~"))
        {
            File.WriteAllText(ManifestFileName, JsonSerializer.Serialize(manifest, JsonSerializerOptions));
        }
    }
}

In each of the test projects, I then added a module initializer like this:

public static class MvcTestingAppManifestInitializer
{
    [ModuleInitializer]
    public static void Initialize()
    {
        MvcTestingAppManifestHelper.AddAssemblyToManifest(Assembly.GetExecutingAssembly());
    }
}

[martincostello]

Hmm - I'm not sure why it didn't work for you. I adopted .slnx for all my projects last weekend (where I use WebApplicationFactory extensively), and the snippet I shared was the only thing I needed to change (and only for projects were I already explicitly used UseSolutionRelativeContentRoot() - it Just Worked™ for projects where I didn't. Example of such changes: martincostello/costellobot@8d50ba7

Regardless, since *.slnx is officially supported

I don't think this is technically 100% true yet, as you need to use Visual Studio 2022 17.14 Preview to use .slnx files - 17.13 and earlier cannot open them at all. I imagine it will "graduate" to "official" whenever 17.14 ships as stable.

[kimsey0]

Do you have any tests where the WebApplicationFactory entry point is in the test assembly like in the reproduction case above, not in a separate assembly?

[siewers]

@martincostello I looked through your tests, and it seems like you are testing an actual MVC application with an entrypoint located in a different assembly, as @kimsey0 noted.

You are right that *.slnx isn't officially supported in Visual Studio, but it is supported everywhere else (dotnet).

[ravndal]

We have both cases.

• The variant @martincostello, where WebApplicationFactory<Program> targets a Program.cs file. This works even without explicitly setting UseSolutionRelativeContentRoot(...).
• We also have a WebApplicationFactory<EmptyStartup>, that fails the same way as this @kimsey0 described here. And as far as I know, the internal lookup for "*.sln" is run before we can call the builder.UseContentRoot(...); or builder.UseSolutionRelativeContentRoot(...); Even the WebApplicationFactoryContentRootAttribute is ignored.

[ninety7]

Are there any plans to fix it in dotnet 10? I see a PR was submitted but seems to be stale...

[kimsey0]

I don't think my pull request is merge-ready (see the comment in the pull request about inadvertently matching solution filter files), but I'm happy to revise it if there's agreement on an acceptable fix.

[ninety7]

I can see the challenge. What about using regex?

[kimsey0]

The solution file pattern is passed into Directory.EnumerateFiles which doesn't support regular expressions. But should we move the conversation about the specific implementation approach taken in my pull request into the pull request discussion? That way, we can leave this issue thread for larger questions like if the default is allowed to change or not. 🙂