For breakfast, in the area you're staying, I llike Pigalle at 8th and 48/49th is great, not sure if they do brunch but what i've had there has been solid.
For something fancier, I second Norma's in Le Parker Meridien. The breakfasts tend to be more like elaborate desserts, the PB&C Waffle...