It stems from a perception that you are somehow breaking unspokenrules, are not meeting some perceived standard or are just not good enough for the task at hand.
My view is that it's exactly that culture of unwritten conventions, unspokenrules, nods and winks that has got us into this trouble in the first place, and I've arrived at the conclusion that the Speaker must go.