You can use what is called Activation Policy:
Objective-C
// The application is an ordinary app that appears in the Dock and may
// have a user interface.
[NSApp setActivationPolicy: NSApplicationActivationPolicyRegular];
// The application does not appear in the Dock and does not have a menu
// bar, but it may be activated programmatically or by clicking on one
// of its windows.
[NSApp setActivationPolicy: NSApplicationActivationPolicyAccessory];
// The application does not appear in the Dock and may not create
// windows or be activated.
[NSApp setActivationPolicy: NSApplicationActivationPolicyProhibited];
Swift 4
// The application is an ordinary app that appears in the Dock and may
// have a user interface.
NSApp.setActivationPolicy(.regular)
// The application does not appear in the Dock and does not have a menu
// bar, but it may be activated programmatically or by clicking on one
// of its windows.
NSApp.setActivationPolicy(.accessory)
// The application does not appear in the Dock and may not create
// windows or be activated.
NSApp.setActivationPolicy(.prohibited)
This should hide the dock icon.
See also
- Answer that inspired this one: How to create a helper application (LSUIElement) that also has a (removable) dock icon
- Documentation for
NSRunningApplicationActivationPolicy
. - A related question: “Start a GUI process in Mac OS X without dock icon”.