I've never written a children's book, but when people meet...

I've never written a children's book, but when people meet me for the first time and I say I write books, they invariably reply, 'Children's books?' Maybe it's something about my face.