One of the biggest differences is that REST implies synchronous processing while MQ is more often asynchronous. As you already mentioned, MQ is a way to decouple producer and consumer so that they don’t have to be online at the same time but the system would still be functioning as a whole. If your use case implies a low-latency response (like a user with a browser) you need synchronous semantics and in this case MQ is just a different protocol. If latency is not a concern, or there’s messaging goes only in one direction MQ may be a very viable alternative. One thing MQ provides for free is load balancing (and some level of HA) on the consumer side.
REST is much more flexible in terms of client/server compatibility. You can run a REST client nearly on every platform, which is not the case with MQ. I guess, this is the reason why some people claim MQ is getting obsolete. For this reason MQ is not good for public APIs. However, for communication between two server systems MQ still remains a very good option.
A unique feature of REST is that it allows you to fully mimic the WEB behavior of your resources (hypermedia), i.e. one resource contains a reference to the other and so on. MQ cannot provide anything like that.
Performance may be another concern. I cannot give any exact figures, but MQ tends to have greater throughput.
In some rare cases due to security requirements clients cannot connect directly to the server. In such cases MQ is pretty much the only way to send data to the server.
To summarize, as a rule of thumb I would use REST
- for public APIs or
- where synchronous processing is needed
MQ
- when only a limited amount of server side systems is involved and
- asynchronous processing is acceptable or
- REST performance is not enough