Can I use a Tag Helper in a custom Tag Helper that returns html?

No you cannot. TagHelpers are a Razor parse time feature.

One alternative is creating a TagHelper and manually invoking its ProcessAsync/Process method. Aka:

var anchorTagHelper = new AnchorTagHelper
{
    Action = "Home",
};
var anchorOutput = new TagHelperOutput("a", new TagHelperAttributeList(), (useCachedResult, encoder) => new HtmlString());
var anchorContext = new TagHelperContext(
    new TagHelperAttributeList(new[] { new TagHelperAttribute("asp-action", new HtmlString("Home")) }),
    new Dictionary<object, object>(),
    Guid.NewGuid());
await anchorTagHelper.ProcessAsync(anchorContext, anchorOutput);
output.Content.SetHtmlContent(anchorOutput);