Muffin, if I have got this right, it's a co-worker who has made this remark not a manager?!

I would refrain from doing anything at the moment.  Try not to hang around or gossip with this person.  I would also be wary of the person who repeated the remark - why would anyone do that?  Do you trust the person who told you or are they known for causing trouble also?

It's hard enough to out a bully never mind a co-worker or manager.

Take a note of the incident for now and file it for future reference.