I thought I had my brewing routine down to pretty much perfect, before I saw a Lance Hendrick video where he recommends sorting out faulty beans and getting rid of most of the chaff.
It may seem tedious to do for each cup, but for me it’s worth it. The cleanness and quality of my brews improved drastically, making me enjoy black coffee way more.
If you’re patient enough, give it a try, it pays off!
I don’t home roast, but my understanding is that depending on the roast level (and to some extent the beans) more or less chaff may stay on the bean itself and only be removable after grinding. Basically the lighter the roast the more chaff will still be there.