Does wearing caps (not baseball caps) look weird with spectacles ?

Very often, I realize that guys who wear caps and fedoras do not wear spectacles or gles. But if a guy who wears spectacles wear a cap or fedora, will it be weird on him?