Update
data.table v1.9.6+ now supports OP’s original attempt and the following answer is no longer necessary.
You can use DT[order(-rank(x), y)].
x y v
1: c 1 7
2: c 3 8
3: c 6 9
4: b 1 1
5: b 3 2
6: b 6 3
7: a 1 4
8: a 3 5
9: a 6 6
data.table v1.9.6+ now supports OP’s original attempt and the following answer is no longer necessary.
You can use DT[order(-rank(x), y)].
x y v
1: c 1 7
2: c 3 8
3: c 6 9
4: b 1 1
5: b 3 2
6: b 6 3
7: a 1 4
8: a 3 5
9: a 6 6