It sounds like you are overusing your inside leg. It sounds like you come out of the circle and either are a little to the inside of the track to start with, or your horse wants to drift to the inside. Then you apply inside leg to keep the horse going straight- which he interprets as change.
Complete guess based on your description, but I'd work on cantering straight lines off the rail. Canter only on the quarter line or center line for a while. Then you can't rely on the wall to do the job of your outside aids- and you will learn to use all of your aids to ride straight rather than overusing your inside leg.
And yes, as someone said, if you circle, then start down the long side- begin circling again before you give the horse a chance to swap off. Try to get one more straight stride of good feeling each time you do it until you make it all the way to the corner.
I completely agree with what you are saying after we complete the circle he anticipates continuing the circle so I end up pushing him back towards the straight away and end up doing too much. There isn't a true rail for us to rely on where we ride but we can definitely work on straight more. Thanks!