Use groupby/shift to apply the shift to each group individually: (Thanks to Jeff for pointing out this simplification.)
In [60]: df['beyer_shifted'] = df.groupby(level=0)['beyer'].shift(1); df
Out[61]:
line_date line_race beyer beyer_shifted
Last Gunfighter 2013-09-28 10 99 NaN
Last Gunfighter 2013-08-18 10 102 99
Last Gunfighter 2013-07-06 8 103 102
Paynter 2013-09-28 10 103 NaN
Paynter 2013-08-31 10 88 103
Paynter 2013-07-27 8 100 88
If you have a multiindex, you can group by more than one level by passing a sequence of ints or level names to groupby's level parameter.