Not sure if or what you're hunting, but 50 yards is pushing "ethical range" for most game.
That being said, the accuracy with a broadhead is not nearly as easy to get as it is with field points. A broadhead that is slightly "out of tune" will fly farther off course as distance increases. With a field point a 1" group at 25 yards should give you a 2" group at 50. With a broadhead a 1" group at 25 yards could easily turn into a 10" group at 50, as the broadhead steers the arrow off course.
I wouldn't begin to guess what your problem is, but I'd suggest that the problem may take a lot of experimenting to solve.
A few thing to try might be:
* mechanical broadhead (much more like a field point in flight)
* heavier broadhead
* different brand of broadhead
* adding weight to the front of the arrow (weights installed in the insert or heavier insert)
* bigger fletching (even switching to 5" high profile feathers would be a possibility)
* square the nock and carefully "spin test" each arrow
I'm sure other suggestions will be added.
Good luck in solving the problem!