Because our atmosphere causes “scattering” of hues in the highest frequencies. This is the same reason the sky appears blue.

Blue light gets scattered more by the atmosphere. So less blue light is received directly in a straight line from the sun and more is reflected from other parts of the sky. That’s also why the sky looks blue in the day and why the evening sky looks red (if the sun is very low during the evening, the blue light can’t reach you because it scattered so much due to the very long shallow way through the atmosphere)

During the day it’s white, but it’s also overhead and blindingly bright so we don’t spend much time looking at it. As it gets closer to the horizon Rayleigh scattering begins filtering out the bluer light and the sun becomes yellow, then orange, then red. It also gets closer to our eyeline and becomes mildly safer to look at so we look at it a lot more. This in turn leads us to believe it’s always yellow