I only had some framing Ls to kinda hold it in place, but I think getting those rubber caps is probably a better choice than going with the square PVC.
But I did find these for the square PVC, I wonder if they would work.
http://www.mocap.com/plastic-caps-square-vinyl.html
or these
http://www.mocap.com/tube-plugs-square-standard.html
those caps aren't for watertightness.
they are just to protect people from cutting themselves on the rough cut ends of the PVC.