I dug into the source code and learned how Schema class is converted by the SchemaFactory.createForClass method.

Well so how it works?

1. Take a look at this example below:

@Schema()
export class Cat extends Document {
  @Prop()
  name: string;
}
export const catSchema = SchemaFactory.createForClass(Cat);

Basically, when you do SchemaFactory.createForClass(Cat)

Nest will convert the class syntax into the Mongoose schema syntax, so in the end, the result of the conversion would be like this:

const schema = new mongoose.Schema({
    name: { type: String } // Notice that `String` is now uppercase.
});

2. How does the conversion work?

Take a look at this file: mongoose/prop.decorator.ts at master · nestjs/mongoose · GitHub

export function Prop(options?: PropOptions): PropertyDecorator {
  return (target: object, propertyKey: string | symbol) => {
    options = (options || {}) as mongoose.SchemaTypeOpts<unknown>;

    const isRawDefinition = options[RAW_OBJECT_DEFINITION];
    if (!options.type && !Array.isArray(options) && !isRawDefinition) {
      const type = Reflect.getMetadata(TYPE_METADATA_KEY, target, propertyKey);

      if (type === Array) {
        options.type = [];
      } else if (type && type !== Object) {
        options.type = type;
      }
    }

    TypeMetadataStorage.addPropertyMetadata({
      target: target.constructor,
      propertyKey: propertyKey as string,
      options,
    });
  };
}

Here you could see what the Prop() decorator does behind the scene.
When you do:

@Prop()
name: string;

Prop function would be called, in this case with no arguments.

const type = Reflect.getMetadata(TYPE_METADATA_KEY, target, propertyKey);

Using the Reflect API, we can get the data type that you use when you do name: string. The value of type variable is now set to String. Notice that it’s not string, the Reflect API will always return the constructor version of the data type so:

• number will be serialized as Number
• string will be serialized as String
• boolean will be serialized as Boolean
• and so on

TypeMetadataStorage.addPropertyMetadata will then store the object below into the store.

{
    target: User,
    propertyKey: ‘name’,
    options: { type: String }
}

Let’s take a look at the: mongoose/type-metadata.storage.ts at master · nestjs/mongoose · GitHub

export class TypeMetadataStorageHost {
  private schemas = new Array<SchemaMetadata>();
  private properties = new Array<PropertyMetadata>();

  addPropertyMetadata(metadata: PropertyMetadata) {
    this.properties.push(metadata);
  }
}

So basically that object will be stored into the properties variable in TypeMetadataStorageHost.
TypeMetadataStorageHost is a singleton that will store a lot of these objects.

3. Schema generation

To understand how the SchemaFactory.createForClass(Cat) produce the Mongoose schema, take a look at this: mongoose/schema.factory.ts at master · nestjs/mongoose · GitHub

export class SchemaFactory {
  static createForClass(target: Type<unknown>) {
    const schemaDefinition = DefinitionsFactory.createForClass(target);
    const schemaMetadata = TypeMetadataStorage.getSchemaMetadataByTarget(
      target,
    );
    return new mongoose.Schema(
      schemaDefinition,
      schemaMetadata && schemaMetadata.options,
    );
  }
}

The most important part is:
const schemaDefinition = DefinitionsFactory.createForClass(target);. Notice that the target here is your Cat class.

You could see the method definition here: mongoose/definitions.factory.ts at master · nestjs/mongoose · GitHub

export class DefinitionsFactory {
  static createForClass(target: Type<unknown>): mongoose.SchemaDefinition {
    let schemaDefinition: mongoose.SchemaDefinition = {};

  schemaMetadata.properties?.forEach((item) => {
    const options = this.inspectTypeDefinition(item.options as any);
    schemaDefinition = {
    [item.propertyKey]: options as any,
      …schemaDefinition,
    };
  });

    return schemaDefinition;
}

schemaMetadata.properties contains the object that you stored when you did TypeMetadataStorage.addPropertyMetadata:

[
    {
        target: User,
        propertyKey: ‘name’,
        options: { type: String }
    }
]

The forEach will produce:

{
    name: { type: String }
}

In the end, it will be used as the argument to the mongoose.Schema constructor mongoose/schema.factory.ts at master · nestjs/mongoose · GitHub:

return new mongoose.Schema(
    schemaDefinition,
    schemaMetadata && schemaMetadata.options,
);

4. So to answer the question:

What should you put as the Prop() argument?

Remember when Nest does the forEach to generate the Mongoose Schema?

schemaMetadata.properties?.forEach((item) => {
  const options = this.inspectTypeDefinition(item.options as any);
  schemaDefinition = {
    [item.propertyKey]: options as any,
    …schemaDefinition,
  };
});

To get the options it uses inspectTypeDefinition method. You could see the definition below:

private static inspectTypeDefinition(options: mongoose.SchemaTypeOpts<unknown> | Function): PropOptions {
  if (typeof options === 'function') {
    if (this.isPrimitive(options)) {
      return options;
    } else if (this.isMongooseSchemaType(options)) {
      return options;
    }
    return this.createForClass(options as Type<unknown>);   
  } else if (typeof options.type === 'function') {
    options.type = this.inspectTypeDefinition(options.type);
    return options;
  } else if (Array.isArray(options)) {
    return options.length > 0
      ? [this.inspectTypeDefinition(options[0])]
      : options;
  }
  return options;
}

Here you could draw the conclusion that:

1. If the options is a function such as String or a SchemaType it will be returned directly and used as the Mongoose options.
2. If the options is an Array, it will return the first index of that array and wrap it in an array.
3. If the options is not an Array or function, for example, if it’s only a plain object such as { type: String, required: true }, it will be returned directly and used as the Mongoose options.

Answer

So to add a reference from Cat to Owner, you could do:

import { Prop, Schema, SchemaFactory } from '@nestjs/mongoose';
import { Document, Schema as MongooseSchema } from 'mongoose';
import { Owner } from './owner.schema.ts';

@Schema()
export class Cat extends Document {
  @Prop()
  name: string;

  @Prop({ type: MongooseSchema.Types.ObjectId, ref: Owner.name })
  owner: Owner;
}

export const catSchema = SchemaFactory.createForClass(Cat);

As for how to add a reference from Owner to Cat, we could do:

@Prop([{ type: MongooseSchema.Types.ObjectId, ref: Cat.name }])

Update

To answer the question in the comment section about:

How to embed schema in another schema?

If you read the answer properly, you should have enough knowledge to do this. But if you didn’t, here’s the TLDR answer.

Note that I strongly recommend you to read the entire answer before you go here.

image-variant.schema.ts

import { Prop, Schema, SchemaFactory } from '@nestjs/mongoose';

@Schema()
export class ImageVariant {
  @Prop()
  url: string;

  @Prop()
  width: number;

  @Prop()
  height: number;

  @Prop()
  size: number;
}

export const imageVariantSchema = SchemaFactory.createForClass(ImageVariant);

image.schema.ts

import { Prop, Schema, SchemaFactory } from '@nestjs/mongoose';
import { Document } from 'mongoose';
import { imageVariantSchema, ImageVariant } from './imagevariant.schema';

@Schema()
export class Image extends Document {
  @Prop({ type: imageVariantSchema })
  large: ImageVariant;

  @Prop({ type: imageVariantSchema })
  medium: ImageVariant;

  @Prop({ type: imageVariantSchema })
  small: ImageVariant;
}

export const imageSchema = SchemaFactory.createForClass(Image);