Game dependent. Most annoying thing I see with the DP4a version of XeSS is ghosting. With FSR, image stability/sizzle/shimmering and ghosting can be annoying as hell at lower resolutions. In most the games I have tried both e.g. Spidey games, cyberpunk, Hogwarts, AC: Mirage, XeSS IQ issues are less distracting. In the Spidey games Insomniac's IGTI is usually better than both for IQ overall. All my personal preferences of course.
Hardware XMX XeSS looks and runs better on ARC than the software fallback on other IHVs, of course.
Sounds like you are fully CPU limited then. FSR balanced vs native should be a significant fps boost at the expense of IQ.
FSR 4 is better than anything but DLSS transformer.