-vf "scale=640:-1"
works great until you will encounter error
[libx264 @ 0x2f08120] height not divisible by 2 (640x853)
So most generic approach is use filter expressions:
scale=640:trunc(ow/a/2)*2
It takes output width (ow), divides it by aspect ratio (a), divides by 2, truncates digits after decimal point and multiplies by 2. It guarantees that resulting height is divisible by 2.
Credits to ffmpeg trac
UPDATE
As comments pointed out simpler way would be to use -vf "scale=640:-2".
Credits to @BradWerth for elegant solution