How to refer to enclosing type definition recursively in OpenAPI / Swagger?

Your definition is perfectly fine. It’s a known issue with rendering recursive schemas in Swagger Editor and Swagger UI:

https://github.com/swagger-api/swagger-ui/issues/3325

To work around the “Example Value” showing null/"string"/undefined instead of a recursive element, you can add a custom example to your schema:

definitions:
  TreeNode:
    type: object
    properties:
      name:
        type: string
        description: The name of the tree node.
      children:
        type: array
        items:
          $ref: '#/definitions/TreeNode'
    example:
      name: foo
      children:
        - name: bar
        - name: baz
          children:
            - name: qux