When Co-workers do things like ask about weekend plans, chat about non-work topics, eat lunch in the same room, they are--subconsciously--reaffirming that they are part of a cooperative (or, minimally, non-antagonistic) social group.
The other primates cement social bonds by grooming each other; we do it by making small talk.
If they solicit your participation in these rituals, and you repeatedly refuse those bids, you are marking yourself out as, at best, an outsider to the group, and thus potentially antagonistic.
This is all happening on the monkey-brain level; they have no idea what they're doing or how they are interpreting your response, so there's no way to clear up the misunderstanding.
To the ape sleeping in your co-worker's DNA, either you are part of the grooming circle, or you are an outsider who, for all it knows, may be coming to steal all the bananas.
Even if you would prefer not to socialize with your co-workers, it's generally worth it to set aside 5 minutes a couple times a week for phatic communication. You don't have to answer your co-workers' affiliative signals every time, but it's less trouble in the long run if you respond to a few of them.