There isn't any way to customize how the price modifiers work. Generally modifiers can only add or subtract the price, not multiply or use any other formula (because is leads to issues in which order the modifiers are applied, among other reasons).
Often the solution to this is to simply "flatten" your modifiers into a single select/radio menu, for example:
- Paper (10 x 10) - $1
- Paper (10 x 10) - $2
- Cardboard (10 x 10) - $5
- Cardboard (20 x 20) - $20
Another option would be to write a PHP extension using the store_cart_item_update_end
hook, which allows you to change the price of items in the cart based on any criteria you need (for example, you can let users enter custom dimensions, then change the price of the item based on the dimensions entered).