OpenAPI 3.0 – how to define file download operation correctly

I believe content is what you’re looking for. You can also use contentReference to reference a reusable object in components.
For example:

paths:
  /files/{fileName}:
    get:
      summary: download file
      operationId: downloadFile
      description: this API is for file download
      parameters:
        - in: path
          name: fileName
          schema:
            type: string
          required: true
          description: The name of file to download
      responses:
        200:
          description: Operation performed successfully.
          content:
            application/pdf:
              schema:
                type: string
                format: binary

components:
  contentTypes:
    application/pdf:
      schema:
        type: string
        format: binary