The channel entries call you are using to retrieve the custom field would need some more parameters to work - you need to give it some sort of channel or entry information for EE to work out what custom field value to return. Also, the syntax for field names requires them to be unbroken strings, so using the form search:custom field= will not work as custom field is not a valid field name.
Nesting channel entries calls like this is better done either using some mechanism to fetch and store the output before you run the main channel entries call, or by using the embed system (though this generally has more overhead, and so best avoided by using a store / retrieve solution if you can).
A simple way to do this would be to use a layout variable to store the custom field value before you construct the channel entries call with search term.
To save the value you'll need to add code a bit like this in a template
{layout="some_group/some_template"}
{layout:set name="some_variable"}
{exp:channel:entries [some parameters]}
{custom_field}
{/exp:channel:entries}
{/layout:set}
and then this code in some later template in the chain (see note below)
{exp:channel:entries
dynamic="no"
search:custom_field"{layout:some_variable}"}
... add some variables etc ...
{/exp:channel:entries}
N.B. You don't say which version of EE you are using - would help generally to add this to your questions as answers vary according to version in some cases.
To save the custom field to a layout variable means chaining at least two templates together, as layout variables are not available for use on the template they are saved on, only in later templates in the chain. So you may need to create a short template simply to do the saving, and then link to your main template from there.
The docs on layout variables are pretty good - so you should be able to work it out from there, but if you hit problems ask some more...!
HTH