IMO, JRPGs are all about 2 major things:
Story
Characters
Everything else can be refined and honed to truly make the game exceptional, but without those two pillars, the game is nothing.
If I find myself groaning out loud, questioning what in God's name the rationale was for a particular plot path, or basically thinking to myself, "Why the $^#@ did THAT just happen?!?", that's a bad sign. That's essentially what happened with FF8, FF9, FF10, and many later, edgier JRPGs.
Also, if I find myself rolling my eyes at certain characters and their dialogue, and basically thinking, "STFU and get off the screen, already" (see FF8, FF9, FF10), that's another failure.
EDIT: (Note: Just to be clear, that's bound to happen a little bit with most Japanese games due to cultural differences. But to be specific, I meant that happens multiple times)
Battle systems are nice, balancing and difficulty/gameplay are important, and I love me some great JRPG music, but that's all secondary to the story and the characters.
I didn't even put graphics down. I honestly think it's the least important aspect of an RPG.
That's why the great, simple 16-bit RPGs of yesteryear blow the ever-living CRAP out of what's come out the last decade: Engaging stories, and characters you care about.
No No No No. You clearly have a rosy memory of the 16 bit era. I still love playing the 16 bit games (FF3/6 again!) but one thing I've noticed is that the development of the characters are a lot more primitive. Intros are somewhat rushed, you are told the history of a character and instantly are supposed to connect with them.
I'll give a good example of a character from FF6, which is a game that receives acclaim for its story: Cyan.
When I first played him, I remember thinking of a valiant and honorable knight (to where his chivalry made him somewhat awkward) whose castle was wrongly poisoned; he fought with his heart for his family, but lost them despite his bravery and had to say goodbye forcefully to them. Only later he had the opportunity to remember the past, do justice for his people, and truly express how much he cared about his wife and children.
That is how I remembered it...until I went back to play the game and then each part felt somewhat dry; the transitions between the emotional scenes also occurred abruptly and felt shallow.
Its still a great game, and some of the dialogue is hilarious (early encounters between locke and celeste are great!) but the character development leaves something to be desired. I think its that creativity played a bigger role in our minds, and we took that with us.
I thought ff9 was okay, but 9 did a MUCH better job developing the character and really letting you feel who he or she was. I got to understand Zidane, Adelbert, Eiko (etc) a lot better than Locke, Celeste, Terra. If you go back and play both, paying attention to the character development, you will start to see that. My main complaint of FF9 is that while you really got to feel for the characters, they actually didn't develop or evolve too much over the course of the game - this is a big problem with FF games in general.
Its one reason why I liked FF8 a lot - Squall and Rinoa's relationship developed slowly and went back and forth at periods of time, each character (well most lol) underwent changes that showed them to be truly different people. Squall at the beginning was not squall at the end; you went from an emo teenager trying to get away from it all to someone who was mature enough to stand up to...a last minute boss that had less than 5 extremely subtle references in the game. In FF7 you saw it somewhat with Cloud...but most everyone was the same: Cid was Cid, Barret was Barret, Tifa was Tifa.
Heck, the whole 'sepiroth=evil' thing was just something that you had to accept until you could go back to the game.
When I think of the FF game that had the best story, the best character development, and the most dynamic characters (I haven't played 12, X-, or 13) I would hands given give it to Final Fantasy Crisis Core for the PSP.