I actually used the a horse a lot, probably because I preferred not to fast travel so often, but Geralt's sprint time was so slow. The horse controls certainly suck(ed) and it's frustrating at times, but I take that over the slowness of walking. That, and I'd rather not fight everything that pops up.
Mounted fighting is quite rad, though, and rather effective, though it's difficult to connect. I had some fun early in the game trying to take on a bandit camp where they were 10+ levels above me, and at least a dozen of them. The only way I succeeded was opening by rushing in on the horse and cleaving a few in half before starting the fight. You can actually catch them off guard this way, as they'll be sitting down, initially and it takes some seconds for them to get up an prepare. ....now that I realize I could have tossed a Northern Wind in there first...damnit!