I went over to Redactor to try the editor out but the standard 'link' button behaves differently to how you describe. When I click the link icon it drops down with "Insert Link" and "Unlink".
Incidentally, if I select "Insert Link" I get 2 text boxes for "URL" and "Text" and a checkbo for "Open link in new tab", if I enter URL:mailto:me@my.com
Text:me@my.com
I get a mailto link exactly as I wanted!
Perhaps you could try adding a "URL" link, and for the URL entering the full mailto syntax, does that give you a decent link??
Also, can I ask what Redactor plug ins are loaded in with Editor? I don't have a licence to download Editor from DevDemon so I can't see for myself! Perhaps its loading the PredefinedLinks or another custom plugin to allow for EE integration (Site page for router pages, ect)?