I think I’ve settled on the latter. Disagreement is maybe best communicated by the absence of an upvote? And downvotes work best when they signal something that is just off base, and while not reportable, is not appreciated at a broad cultural level.
I’ve upvoted comments that I disagreed with, but were well written an contributed to a good discussion. I only downvote for very low quality, spam or hateful comments.