INB4 "Compressing is the devil!!!!"
Welcome to the wild, wild west. Where facts and absolutes don't exist and what works best for you may be useless to someone else. I can only tell you what works best for me.
Do you understand the difference between containers and codecs?
.vob and .mkv are containers. Think of them as just like tupperware. Other common container types are .mp4, .avi, .m4v, .mpg, .mov, etc.
The containers (.mkv, .mp4, .vob) hold audio, video and subtitle streams like .h264, .divx, .ac3, .pgs, .srt, .aac, etc. Certain containers are more flexible than others. .vob containers are a DVD standard and accept an MPEG-2 video codec and an ac3 and/or dts audio stream plus any subtitle streams. You could unpack those streams and put them into an .mkv and the quality will be exactly the same. Just like if you take macaroni out of a bowl that says .vob and put it in a different shaped bowl that says .mkv. It's the same macaroni.
When you shrink a .vob file, you are transcoding the MPEG-2 video stream and the .ac3 or .dts audio stream and reducing the bitrate. This is where the quality goes down. If you have a 6GB .vob and want it to trancode it to a DVD-5 .vob you are just removing information from the streams to make them smaller. Some material compresses quite well, some does not. Simple animation compresses very well because of the lack of detail, but action movies with lots of dark scenes do not compress well at all. You get artifacts as the codec tries to guess about what should be displayed and your dark scenes get blotchy and blocky.
.h264 compresses much more efficiently than MPEG-2 at lower bitrates. A 4GB .mkv with an .h264 stream can be of much greater quality than a 4GB .vob. By the same token, .aac compresses much more efficiently than .ac3. You can make the .mkv as large or small as you'd like.
Size of playback screen is the biggest factor. If you are on 110" projection, then ANY compression will probably be noticeable. If you are watching a 32" from 10 feet away, then any reasonable compression will look the same. Keep in mind, with DVDs you are are also getting upconversion if you display it on an HD TV. The more info the upconverting device has to work with, the better the upconverted image will be.
.iso is a container type, but it is just basically packing a photograph of the original into the .iso file. It will take anything but requires special and unpacking.
If you want to keep all of the menus and goodies then an uncompressed .iso is the best option. The .iso will run just like you stuck the DVD in the player with the exact same quality.
If you just want the movie and space is important, the size of the TV will be the biggest factor. On 110" projection, any compression will be visible. If it is a 32" that you watch from 9 feet away, you can shrink the movie itself down to about 1GB with an .h264/.mkv and probably won't notice the difference.
I personally have 1 DVD player in the house that hasn't been used in 3 years and the next menu or DVD extra that I watch will be the first one. I don't anticipate DVDs making a comeback either so I eliminate any necessity for .iso or .vob.
I do not have a $10,000 Dolby THX certified audio setup on the main tv. I do have 4 tvs from 17" to 50" connected to a combination of PS3, WDTV Live and DirecTV receivers that playback my movies stored on a central computer. The kids also like to watch movies on the 7" Kindle Fire. Occasionally, there is a need for the movie to be on another portable device like an Android phone, so I try to make my encodes compatible across the entire field if possible. So, while .mkv is the most flexible format, not every device supports it so I go with .mp4 or .m4v.
I rip Movie Only from the DVD or Blu-Ray as an .m2ts or .vob file using DVD Shrink (DVD) or Tsmuxer (BR), keeping the original video stream and the 6-channel .ac3 or .dts audio stream. I run them through Handbrake and use .h264:
DVD - 1200-1500kbps
720p - 2200-2500kbps (Most of my HD rips)
1080p - 6500kbps (My favorite handful of movies)
I insert 2 audio streams. Some devices (like an Xbox and lots of portable devices) will insist on a 2-channel .aac as the first audio stream or the file will just crap out so the first is always an .aac 2-channel at 128kbps and the 2nd is usually a 6-channel .ac3 at 448kbps or 640kbps if I think the extra bitrate might be needed in the future. I might keep the .dts instead of an .ac3, but I'm back and forth. I use no more than 4 reference frames, the PS3 will choke on anything more than 4. I also set 3 b-frames. The rest of the settings are dependent on how fast I need the encode to be done versus how much that last 10-20% of quality matters to me.
Ultimately, your best bet is to just tinker with it and find what works best for you.