Whether or not I am annoyed depends purely on how important the conversation is. If somebody talks sh*t about nothing and I can't escape (e.g. in a train) I feel like I need to murder somebody. But if there is point to a conversation and I feel it is important I don't care and I switch off completely regardless of how boring the subject is.
The most annoying thing in my life right now is a colleague in my office, who talks to his two year old daughter in child's voice about things children talk when she phones him. I immediately go and leave my desk or put my headphones on otherwise there will be blood

My "big boss" sits two desks away from me and he talks about boring stuff (financial accounting) all day long on his conference calls but I don't mind at all. I know it is important and I just block it in my head..