I don't know about the downsides, but the last fem seed line I crossed; the first 5 seeds I popped were female. It was females seeds Neville's haze hybrid with a male hashpassion from seedsman.
The downsides only exist in peoples minds. Look at all the elite plants that are fems. As far as the plant that you're gonna see smell touch and taste, there is none.