I think you need to look at the code you are using for the add to cart bit.
Normally, with CartThrob, you would use {exp:cartthrob:add_to_cart_form} to do the "buy now" bit.
If you look at the docs for this tag pair...
http://cartthrob.com/docs/tags_detail/add_to_cart_form/index.html
You will see that you can specify a quantity parameter in the tag pair.
However, you can also add a form field to allow for custom quantities - i.e. you can add more than 1 product at a time to your cart.
I may well be wrong, but I have a feeling you have both the quantity parameter AND a quantity field. Just a hunch. To be honest I don't know what would happen if you did but it's my best guess with the info so far.