I think the powers that be have more important things to keep track of than using customs to track down individual hobby growers. Still, your good name and address is now tarnished. I'd be more worried if I lived in a small town and the folks in the local post office were talking to each other and their spouses about your package that had been intercepted as contraband. Rent a private PO Box where nobody knows you.
good advice, don't use small town post offices, everyone knows everyone... a big, busy post office is best...