You can select the option "None" in the fallback user field. Choosing this value is the same as choosing nothing/leaving it blank.
Also, it should be possible to select a fallback user "unassigned", so that if no matches are found for users, the field would be set to "unassigned".