Messaging tends to concentrate on the reliable exchange of messages around a network; using queues as a reliable load balancer and topics to implement publish and subscribe.
An ESB typically tends to add different features above and beyond messaging such as orchestration, routing, transformation and mediation.
I’d recommend reading about the Enterprise Integration Patterns which gives an overview of common patterns you’ll tend to use in integration problems which are all based above a message bus (though can be used with other networking technologies too).
For example using open source; Apache ActiveMQ provides a loosely coupled reliable exchange of messages. Then you can use Apache Camel to implement the Enterprise Integration Patterns for smart routing, transformation, orchestration, working with other technologies and so forth.