Just a thought sour, but if you use the next size up from your fan, insulated duct, it will quiet things down quite a bit (i.e attach 8" insulated duct to your 6" fan via reducers). The larger duct area and insulation helps reduce the whoosh sound considerably.
My 8" fan is practically silent with 8" R8 insulated duct. The 6" is dead silent. I don't roll with speed reducers... but to each their own.