Plugins as you knew them from Rails 2 (i.e. plugins under the vendor/plugins folder) were deprecated for Rails 3.2; support for it was completely removed in Rails 4. Now, there’s a concept of a “gamified plugin” where the plugins are essentially built as gems and can be shared across different Rails applications.
But to answer your question about gems vs plugins, check out this Stackoverflow answer. Long story short, plugins from the Rails 2 universe is an extension of the rails application, whereas a gem is a packaged ruby application.
As for Rails engines, I’ve found this to be a pretty easy and intuitive definition of a Rails engine:
Rails Engines is basically a whole Rails app that lives in the container of another one. Put another way, as the docs note: an app itself is basically just an engine at the root level. Over the years, we’ve seen sen engines as parts of gems such as devise or rails_admin. These examples show the power of engines by providing a large set of relatively self-contained functionality “mounted” into an app.
And since both rails engines and plugins are types of ruby applications, they can all technically be packaged and used as a gem (usually).