Pass ILogger or ILoggerFactory to constructors in AspNet Core?

This is more of a design issue.

The controller should not be creating child classes anyway. That is not a concern the controller should be dealing with and goes against SRP (Single Responsibility Principle).

My question is what should I pass to child classes called from my controller

If your preference is to separate the loggers, then there really isn’t any other choice here than to have child (dependent) classes have their own loggers.

Have child classes inject their own logger

public class TodoRepository : ITodoRepository {
    private readonly ILogger logger; 

    public TodoRepository(ILogger<TodoRepository> logger) { 
        this.logger = logger;   
    }
  
    //...
}

and then inject the child class into the controller.

public class TodoController : Controller {
    private readonly ILogger logger;
    private readonly ITodoRepository todoRepository;

    public TodoController(ILogger<TodoController> logger, ITodoRepository todoRepository) {
        this.logger = logger;   
        this.todoRepository = todoRepository;
    }

    //...
}

That way, the child loggers will get resolved when the child classes are being resolved and injected into the controller.