I've tried to remove all the male flowers from plants that were slightly hermed, and never been successful. You will have seeds if you have a herm, from my experience.
The thing is, the seeds will carry the hermaphroditic trait, Though I don't think this holds true if the plants hermed as a result of light stress, as it sounds like. But if they are the result of a genetic hermie, that trait will be passed on even though it is a recessive gene.
If it is your only plant, keep it, but if it is gonna pollinate things, I would take it now and be done with it. PEace...