Stop the presses,
I don't see in your question if you followed the documented process over at bold minded here : https://boldminded.com/add-ons/wyvern/documentation (all my quotes below are from this site)
So Step one (which I assume you've done as you can see it in the field list) :
Adding 3rd Party Plugins (after version 1.2.7)
Place your CKEditor plugin inside the
/themes/third_party/wyvern/plugins/ folder. In the ExpressionEngine
control panel go to Add-ons » Fieldtypes » Wyvern and select your
plugin in the multi-select menu to enable it.
Now, Step 2 all you need to do is add it to the toolbar :
Adding buttons to the toolbar
By default, Wyvern intentionally comes with a limited set of CKEditor
buttons in the Fieldtype settings page. I have chosen to keep the
default list to the essentials for simplicity. If a button you need is
not found in the Toolbar list, open the
system/expressionengine/third_party/wyvern/config.php file. You will
see an array which defines the default list, as well as the rest of
the available CKEditor buttons, which are commented out. To enable a
button, simply remove the code comment prior to that button. Go back
to the Add-ons > Fieldtypes > Wyvern page and the button will be
available within the Toolbar list.
Note This is not CKEditors "config.js" but Wyverns "config.php".
I'm not sure from your question if you have seen this page or anything, but can you try this process and let us know how you get on! I would recommend rolling your changes back and trying this first...
config.extraPlugins = 'youtube';
line getting missed for the config for the specific editor? I remember this from trying to add the ImageMap plugin to CK on Wygwam. – Blatant Apr 14 '15 at 10:21