'They never remember that I'm at the dinner table too,' he said, sitting alone (with me) at a big gathering of the bank as other top executives were toasting from one table to the next.
One way of understanding this is to imagine that you are traveling with a speaker of such a language and staying in a large chain-style hotel, with corridor upon corridor of identical-looking doors.