He was hurt when people didn't either embrace Yoko or his love for Yoko, but finally he didn't care.
People hated him for leaving the Beatles or for falling in love with Yoko ... they wanted him to be a Beatle and he fought that all his life.