First off, if you’re worring about binary size, you don’t really need arm64-v8a, all those devices can run the armeabi-v7a binaries just fine. Only if you really need to cram the last extra performance out of it, it might be worthwhile.
As for armeabi and ARMv6; Android itself doesn’t officially support it any longer, since Android 4.4 (October 2013) – and since Android 4.0 it should be much less common (from that version, AOSP source requires modifications to still build for ARMv6). So in practice, if you aren’t supporting versions below 4.4, you can drop that one without any significant loss.
Also, for x86; many of those devices ship with surprisingly decent emulation of arm binaries, so those can manage with the armeabi-v7a version just fine as well.
EDIT: The above was written in 2015; these days Play Store requires that apps include support for arm64-v8a. But these days the next question is more about whether you need to include armeabi-v7a at all, or if the market share of 32 bit devices is small enough to drop support for.