Of course you should! It is very high in silica, an immune booster. Also contains calcium and something else,(lime?) I believe. Basically the same thing as hygromite, but finer ground.
It will only kill bugs once it has dried out and will not hurt worms.