The roughest part is showing up. Once you throw yourself into the scene, it's just great fun to let it all go and not be self-conscious, and stop questioning whether you're sufficient.
Another mistake a director can make is not to be prepared, so you get there on the day to shoot the scene, and they don't know how it should be blocked, and they're not clear on how they want to do a scene.
It's hard on an actor when you have to do a scene 45 times and you know damn well that three of the angles a director is shooting will never make it into the movie.