The female voice is actually more complex than the male...

The female voice is actually more complex than the male voice, due to differences in the size and shape of the vocal cords and larynx between men and women, and also due to women having greater natural 'melody' in their voices. This causes a more complex range of sound frequencies than in a male voice. When men hear a male voice they process it in the 'mind's eye.' This is the part of the brain where people compare their experiences to themselves, so the man is comparing his own voice to the new voice.