I think you’ve fundamentally misunderstood the idiom. In what scenario would someone be telling this idiom directly to you in regards to your appearance?

This idiom is typically applied generally about the subjectivity or art, and occasionally in the defense of a particular piece of art on the grounds of subjectivity.

It isn’t used to cast judgement on the appearance of a person, except maybe ironically/sarcastically.