Every set has their own personalities and their own quirks. It's funny comparing starring in a show to going in and doing recurring work - there's already a rhythm that's been established, and people know each other so well. It's like being a guest in someone else's home.
Honestly, acting is the most work when you're unemployed. For me, the actual acting part is never hard. It's the politics and basically everything around the acting that is difficult.