Im 6'4 and the large fits, so yes, go with the small.
Its a great jacket for the price, however if I were you, spend a little more and get something that breaths a little better. If Dianese is out of your price range, look at the Troy Lee suits. About half the price as Dianese, and they are supposed to be just as breathable, as they are close to the same design/materials.
The Dianese suit is awesome. I will never by a different suit again. Well worth the money.