MySQL use HAVING statement for this tasks.
Your query would look like this:
SELECT g.group_id, COUNT(m.member_id) AS members
FROM groups AS g
LEFT JOIN group_members AS m USING(group_id)
GROUP BY g.group_id
HAVING members > 4
example when references have different names
SELECT g.id, COUNT(m.member_id) AS members
FROM groups AS g
LEFT JOIN group_members AS m ON g.id = m.group_id
GROUP BY g.id
HAVING members > 4
Also, make sure that you set indexes inside your database schema for keys you are using in JOINS as it can affect your site performance.