You can use has():
ul li:has(ul.sub) { ... }
It’s not possible with CSS3. There is a proposed CSS4 selector, $, to do just that, which could look like this (Selecting the li element):
ul $li ul.sub { ... }
See the list of CSS4 Selectors here.
As an alternative, with jQuery, a one-liner you could make use of would be this:
$('ul li:has(ul.sub)').addClass('has_sub');
You could then go ahead and style the li.has_sub in your CSS.