Making them wait until level 4 to start seeing penalties seems a long time to go without. I’d like to introduce it earlier, but I don’t want to do so in a way that seems too punishing at the low levels when every +1 is very significant. I’ll have to think about it some more.
I guess it’s time for me to start compiling my Starfinder 1e shopping list. I have the Core Rulebook, but that’s it.