I've now done loads of tests and have come to the conclusion it's not possible to leave out the template_group from the url unless it is using the default template group.

Each template_group has the following templates: index _home _category _single-entry 404

For info the simplified index template code I'm using is:

{if segment_1 == "category" AND segment_2 !=""}{embed="subsite/_category"}
{if:elseif segment_1 !=""}{embed="subsite/_single-entry"}
{if:else}{embed="subsite/_home"}
{/if}

The .htaccess cose is bog standard:

#remove trailing slash
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} ^(.+)/$
RewriteRule ^(.+)/$  /$1 [R=301,L]
# remove index.php
RewriteCond $1 !\.(gif|jpe?g|png)$ [NC] 
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ /index.php?/$1 [L]

As soon as you add a segment to the url it looks for a matching template in the default template group. So if you set the template group and template in the subsite index.php like:

$assign_to_config['template_group'] = 'subsite';
$assign_to_config['template'] = 'index';

Then visit subsite.org It will load the index template from the subsite template group, but as soon as you add a segment it looks in the default template group for a template name matching the segment and then loads the main sites template, if there is no template name matching the segment it will load the default template group index template (whilst still displaying the subsite url in the address bar).

I tried adding a template to the default template group (subsite-index) and then changed the index.php to:

$assign_to_config['template_group'] = 'mainsite';
$assign_to_config['template'] = 'subsite-index';

Again, it loads the subsite-index template when the url has no segments but as soon as you add one it loads the mainsites templates.

The correct domain name is always showing in the browser address bar and plain html files can be called/viewed from either domain.

Studying the James Smith URL schematic closer it does seem to confirm my findings. There is no global variable for default_template_group and the database entry in 'exp_template_groups' has a field 'is_site_default' y or n. Then in numerous system folder files a check is done to establish the template_group, an example from core.template.php is below:

// We we are not enforcing strict URLs, so Let's fetch the the name of the default template group
$result = $DB->query("SELECT group_name, group_id FROM exp_template_groups WHERE a is_site_default = 'y' AND site_id = '".$DB->escape_str($PREFS->ini('site_id'))."'");
// No result?  Bail out...
// There's really nothing else to do here. We don't have a valid template group in the URL
// and the admin doesn't have a template group defined as the site default.