In a custom extension, can I make a settings field required to have a value without having a create a custom form using settings_form()?
Not really, no. You could force front-end validation by crafting an Accessory and making those fields have a "required" attribute, but you're best bet is to just use the
settings_form() functionality. It's not hard, just follow the guide!
As an alternative to making fields "required", you could only set settings that are radio fields or select fields, and have defaults. It's really not the same, but I'm not sure what you're requirements are.