First, identify the duplicates. Second, join back to extract these rows.
A non-aggregated (or non-window/ranking) self join forms a partial cross join and gives the square of duplicates for any set of keys. Including non-duplicates too. 1 x 1 = 1 after all.
SELECT
t2.*
FROM
(
SELECT
StateId, OrderId, OrderTime, PermitId
FROM
myTable
GROUP BY
StateId, OrderId, OrderTime, PermitId
HAVING
COUNT(*) >= 2
) T1
JOIN
mytable T2 ON T1.StateId = T2.StateId AND T1.OrderId = T2.OrderId AND
T1.OrderTime = T2.OrderTime AND T1.PermitId = T2.PermitId