If you only deal with integers, you can use str.isdigit():
Return true if all characters in the string are digits and there is at least one character, false otherwise.
[row for row in listOfLists if row[x].isdigit()]
Or if negative integers are possible (but should be allowed):
row[x].lstrip('-').isdigit()
And of course this all works only if there are no leading or trailing whitespace characters (which could be stripped as well).