sounds like you are applying the power too early in your stroke. Think "lead before speed" and "speed up and stop" to conceptualize the principle of acceleration during the cast.
Here is a video clip that might help. It is a spey cast but you can clearly see that the stroke starts slow and continually gains speed until it stops dead at the point of maximum acceleration. The same principle applies to the single hand cast.
and here is another one: