I guess libertarianism is always considered so weird and fringe that people assume that you're in the closet if you don't go around talking about it.
Never assume that the guy understands that you and he have a relationship.