I mean for those plants the model should be trained to spit out the next highest common denominator / family instead of the specific species.

Most people are going to take photos of the leaves, stem or at best the outside of the flowers. These are rarely conserved within families. You’ll need the arrangement of the four floral whorls to name a family and expect any degree of accuracy. And that’s assuming your plant is an angiosperm.