How can I get NUnit3TestAdapter to work with .Net Standard 2.0?

You don’t provide enough information to diagnose what is wrong with your project file, but it 3.9.0 of the NUnit Adapter does work with .NET Standard 2.0 and F#. I think that your test project is targeting .NET Standard. It needs to target .NET Core or .NET 4.6.1+. Test projects are treated like executables, they need to target a specific framework, not .NET Standard. The code that you are testing can be .NET Standard though.

You should follow the documentation Unit testing F# libraries in .NET Core using dotnet test and NUnit and read the NUnit docs, .NET Core and .NET Standard.

Your project file should look something like this,

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netcoreapp2.0</TargetFramework>

    <IsPackable>false</IsPackable>
  </PropertyGroup>

  <ItemGroup>
    <Compile Include="Tests.fs" />
    <Compile Include="Program.fs" />
  </ItemGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.5.0" />
    <PackageReference Include="NUnit" Version="3.9.0" />
    <PackageReference Include="NUnit3TestAdapter" Version="3.9.0" />
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="..\MathService\MathService.fsproj" />
  </ItemGroup>

</Project>