Is there a way to set the min order qty for a category? E.G. A customer can only order 5 products in category A; either the same product or 5 different products in the same category.
Not per category as such but you can set this per product on the Product entry/edit page:
Although you can run your own conditional check to see if the customer has at least 5 products of a particular category in their cart but this would need to be done in the cart/checkout as the product page would be a bit pre-mature (they may go on to view other products in the same category and add those to their cart). This wouldn't be great for usability as they'd only find out once they're ready to checkout. If you're going to do things that way then be sure to make a clear note on the product pages or on a global area of the site to try and prevent any annoyances when it comes to the checkout.
You may need to play with the conditional logic to loop through the items, tally up associated item counts and then compare that to categories. It may be a little tricky but certainly possible.