Yep, the main difference is that Literal controls just render out text, but Label controls surround it with <span> tags (Unless you use the AssociatedControlID property, in which case a Label control will render a <label> tag).
So, labels can be styled easier, but if you’re just inserting text, literals are the way to go. Literal controls also have a handy property Mode which governs how the text is rendered. You can have it HTML-encoded, or rendered without any changes, or have any “unsupported markup-language elements” removed.
If you’re not applying any styles (e.g. by using Label‘s CssClass property), it will be fine to replace Label controls with Literal controls.