I think the easiest way would be to make sure that your call to the css file is within your channel entries loop and that you name your css files the same as your channel short names.
{exp:channel:entries limit="1"}
<head>
<link href="path/to/css/folder/{channel}_styles.css" rel="stylesheet" />
</head>
<body>
{title}
{other_channel_fields}
</body>
{/exp:channel:entries}
In this example your css file name would be "blog_styles.css" and your channel would be "blog" for the sylesheet that is shared across multiple channels I would create a stylesheet for each channel shortname then I would @import
another css file so you are not duplicating the styles across 2 templates.
If you are not serving a single entry make sure that you wrap your css file declaration in {if count == 1}
like
{exp:channel:entries}
<head>
{if count == 1}
<link href="path/to/css/folder/{channel}_styles.css" rel="stylesheet" />
{/if}
</head
{/exp:channel:entries}
I should also mention that you could also use the channel name in a conditional to load specific files.
{exp:channel:entries}
<head>
{if channel == "blog"}
<link href="path/to/blogs_css_file.css" rel="stylesheet" />
{/if}
{if channel == "about"}
<link href="path/to/about_css_file.css" rel="stylesheet" />
{/if}
</head>
{/exp:channel:entries}