D-Fend is a good front-end for DOSBox. However, it's more flexible to edit the dosbox.conf yourself, since there is already enough explanation in dosbox.conf.
For the video thing, probably it's about the core and the output, or further, your video driver. Driver is the bane of ATI cards. It's no telling which is which, you have to experiment yourself.
As you have a fast machine and powerful GPU, you may want to give the DOSBox CVS builds a shot. There are a lot of bug fixes and enhancement added since the release of 0.65. The status of playing No Regret in DOSBox on my machine is upgraded from "slideshow" to "barely playable" after I started to use CVS builds recently.
There is one more option "AUTO" for the core and the cycle part, you don't have to bother with the performance thing. All are self-determined. For output, surface seems to be the most stable(and default) method of output. You may also try overlay or ddraw in 0.65, and openglhq or direct3d for high quality video in CVS.
D-fend does not cover the options added in CVS yet, so I edit the confs myself, and use D-fend as a multi-profile launcher.