Politicians cannot show up on the doorstep to Downing Street and say 'I have millions of votes under my arm so now you must arrange a referendum'