i personally wouldnt be spraying anymore neem on your plants, at this point its not going to help much out... i would go ahead and try to shake your plants so they are as dry as possible tonight, go ahead and spray them down in the morning, let them dry out ... and mix a few drops of soap with the BT (whatever product you get that has BT in iT) than spray the plants down(with the BT and soap mixture)... have you not seen any worms, small or large on your plants?
it sure looks like worms , and thats the first thing i would assume would be eating at your buds.