I'm trying to do something like orderby="custom_order"
What hook would let you add and run a custom parameter option?
A hook would probably not be the best way to do this. With a module, you can extend the functionality of the channel:entries tag by passing your custom module tag off to channel:entries.
This gist is an excellent example to follow.
Basically you'd have to use the parameters sent to your custom module to create a pipe-separated list of entry ids that you want in the order you want them. Then you'd feed it to the channel entries tag by using
$this->EE->TMPL->tagparams['fixed_order'] = $entry_ids;
Extending the channel:entries tag like this makes it act exactly like a normal version of the tag, so all the parsing works perfectly, etc.