Rails 5

As of Rails 5, you can signal that an operation should be aborted by explicitly calling throw :abort inside your callback. The documentation section on cancelling callbacks (now) states:

If a before_* callback throws :abort, all the later callbacks and the associated action are cancelled.

The following section on transactions continues:

If a before_* callback cancels the action a ROLLBACK is issued. You can also trigger a ROLLBACK raising an exception in any of the callbacks, including after_* hooks.

Rails 4

The story is pretty similar to Rails 5, except that callbacks should instead return false. The corresponding parts of the documentation helpfully states

If a before_* callback returns false, all the later callbacks and the associated action are cancelled. If an after_* callback returns false, all the later callbacks are cancelled.

Followed by

If a before_* callback cancels the action a ROLLBACK is issued. You can also trigger a ROLLBACK raising an exception in any of the callbacks, including after_* hooks.