Maybe that’s his way of saying people assumed he had a bad relationship with her? I don’t think he meant they literally said “you didn’t like your mom.”