Excellent thread! I love this kind of stuff! But I have a feeling that after you're done, you'll end up with an inferior product that will have cost you more (not only $$, but time & resources) in the end.
Your rail / motor design is nearly identical to that of a Gualala Light Rail 3 / 3.5.
Do you know how they (Gualala) get the motor to turn back? Just a simple switch / actuator tied to a paddle, so that when it hits the stopper the paddle pushes in, flipping the switch.
This is my Light Rail 5, but the actuator / paddle system is identical. Can you see it on the second picture, the plastic paddle on the right side?
I think the Light Rail 3.5 is only ~$100.