I think it's probably doable. It may be a process of elimination to work out all the kinks as you go, but no core hack should be needed.
I'd suggest you move the system directory out of the main site's document root first.
You can remove admin.php from the main site. And copy it over to the admin subdomain. And you can rename it whatever you like.
You'll need to adjust the $system_path variable in the index.php of the main site and admin.php of the admin site to match the new system folder location.
From there, you'll need to make sure the themes directory is accessible to both document roots, through a symlink.
The upload locations will need to be set with the full URL and an absolute path, so both sites can access the files.