VMware and Backups

[AstroCreep]

Good day all,
So I just started a new position at a new company (I'm a 'Junior Network Administrator') and I'm walking into a position where we're looking more & more into disaster recovery. One of the things that we are doing is running a Windows 2003 Server with VMware Server running on it. We will be doing a 'file-level' backup inside each VM via Backup Exec, but we'd like to figure out a way to take a of each VM while they are running, if that makes sense.
That's the feature that is the most attractive to my network admin - quickly being able to 'recover' a VM on to another system if need be.

The issue is that, from what I have found, there is no really Good way of doing this.

I did an experiment and copied the folder that a VM was stored in (while it was running). I then deleted the folder with said VM and replaced it with the 'copy' I made. At first VMware (workstation, mind you) b*tched at me about not having exclusive access to the VM and asked me if I wanted to "Take Control". I clicked yes, but it wouldn't let me until I deleted a locked log-file or something (I can't remember what I needed to delete as I did this a couple of weeks ago, but it was nothing more than a text-file of some sort). After I deleted the file, the VM fired right up.
The only problem with doing that was is that when the VM started back up (another Windows 2003 Server) it threw me one of those messages saying "I was powered down unexpectedly; give me a reason why". Any of us who use Windows Server 2003 and have had it lose power or BSoD & reboot know about the message I'm talking about.

So here I am now - wanting to see if there's a "Good Way" to take a backup of a VM that is running. I checked the VMware forums, but for the most part, no one there had any real good solutions for the free VMware (we are planning on moving to ESX in the next few months though).

Any input is appreciated.
Thanks!

 

[loup garou]

I believe you could run vmware converter inside the VM as if it were a physical machine, but it's not ideal from a scheduling standpoint. I'm sure it could be scripted though...

You can use the free starter version now and purchase enterprise when you go to esx.

 

[Nothinman]

If I was going to use the "copy the files behind the VM's back" method I would at least want to take a snapshot of the volume so that I was sure the files were consistent during the copy and it would be nice if there was a way to tell VMWare to sync the disk files with it's in memory copy but vmware-cmd doesn't seem to have a way to do that and I don't think just running 'sync' (or whatever equivalent you might find on Windows) is enough.

 

[AstroCreep]

Originally posted by: loup garou
I believe you could run vmware converter inside the VM as if it were a physical machine, but it's not ideal from a scheduling standpoint. I'm sure it could be scripted though...

You can use the free starter version now and purchase enterprise when you go to esx.

That sounds like an awful lot of work (not that I'm trying to be lazy or anything though). I actually used the converter while toying around with Backup Exec System Recovery (can take BESR 'Snapshots' and import them into VMware VMs; kinda neat).

In essence, all I really need to do is make a 'copy' of the folder containing the VMs. Scheduling the system to make a copy of a folder is no big deal at all, it's just the potential of having to run a 'backed-up' VM and the issues that may or may not have arisen from the "Hard-booted" VM.

 

[AstroCreep]

Originally posted by: Nothinman
If I was going to use the "copy the files behind the VM's back" method I would at least want to take a snapshot of the volume so that I was sure the files were consistent during the copy and it would be nice if there was a way to tell VMWare to sync the disk files with it's in memory copy but vmware-cmd doesn't seem to have a way to do that and I don't think just running 'sync' (or whatever equivalent you might find on Windows) is enough.

Snapshot of what volume? The 'physical' volume on the 'Host' where the VMs are stationed or the volumes in the VMs?

 

[Nothinman]

Snapshot of what volume? The 'physical' volume on the 'Host' where the VMs are stationed or the volumes in the VMs?

The former, on Linux it's simple to create a snapshot of a LV that stays consistent while the original LV can still be used and changed.

 

[Thor86]

Check this out:

http://communities.vmware.com/thread/54782

 

[Nothinman]

Um that script shuts down the VM before doing the copy...

And it's entertainingly long too, almost 500 lines to just shutdown a VM and zip up the image.

 

[Brazen]

I run ESX Server at work and have had no end of troubles with it's Consolidated Backup. Personally, I think I could right a better script myself and use the free VMWare Server.

First of all, get it running on Linux, not Windows. Linux's LVM will be a big advantage here, plus the OCFS file system is probably the best file system to use for your virtual machine storage (called a "datastore" in eSX). OCFS is almost identical to VMWare's VMFS used on ESX Server is why I would think that.

Now you can script all this from the host server. First have it take a snapshot of all your virtual machines (you could probably even make this dynamic to automatically find any new virtual machines). Then have it flush the datastore partition's disk cache* and take a snapshot of the partition using LVM. Then mount the snapshot somewhere as read-only and have your backup software backup from the mounted. It would even be a good idea to have your script clean up the virtual machine files on the mounted snapshot before backing them up, such as deleting any lock files, so when you restore them they are ready to go.

* one thing I don't know about OCFS is if it is even a command to flush the disk cache. I may consider XFS for this reason because I know it has easy-to-use commands for flushing the disk cache. edit: oops, I would want to run this from an iSCSI cluster, so I can't use XFS. edit2: oh wait, since OCFS is a cluster-away file system, maybe it doesn't use a cache, or at least not the same way.

I'm actually trying to put together a website of server guides and this is one of the first things I want to put up. I know how to do it in my head, but I'm slow at getting things in print. I think a good backup plan is crucial which is why I let EMC (parent company of VMWare) put together our whole backup solution for our VMWare ESX Servers including every piece of backup hardware and software. Almost a year later, EMC's tech support still has not gotten us to where we can restore a virtual machine from tape backup /rant.

 

[AstroCreep]

Here's what's funny - we're probably going to go to ESX within the next year (just a matter of time and planning). But we need something to use now (the IT battlecry, as it were). ...and unfortunately I have never used Linux in my life (and my current 'Push' on the education front is along the MCSA/E path).

And I saw that guy's script, and yes, it does shutdown the VM, but from what I can find, that may be the only way to get away with it. I can't find any other software that can do what I need it to do. I mean, yeah, we'll be doing normal Backup Exec file-level backups, but nothing can beat the ability to just fire up a VM that's saved on the SAN from a different PC (if the need should arise due to a hardware failure).

Hell, I'd even consider a commercial offering if there is one (depending on price, of course). I've tried Backup Exec System Recovery, but that software hasn't proved itself to be worth a damn, let alone $1G.

 

[Nothinman]

If you're going to use that script at the very least change it to suspend the VM instead of shutting it down so you don't have to wait for shutdown/bootup of the VM.

 

[Thor86]

Suspending VMs for services in mid-transactions (ie: databases), is a no-no in VM.

 

[AstroCreep]

Originally posted by: Thor86
Suspending VMs for services in mid-transactions (ie: databases), is a no-no in VM.

Why? Could I Just script net stop/start commands for the services before suspending/pausing the VM?

 

[Nothinman]

Suspending VMs for services in mid-transactions (ie: databases), is a no-no in VM.

One would assume that the backup script is going to be running when traffic is low or non-existant. But even so, I've had no problems suspending and resuming VMs with open TCP connections and having the TCP connection stay open. Hell just yesterday I hibernated my home physical machine via SSH and immediately resumed it and the SSH session continued where it left off when the resume finished.

 

[AstroCreep]

One thing I just read was that you can't run a 'Suspended' VM on a different server, but I couldn't find why.
http://communities.vmware.com/message/407181 - second post (VMware employee, apparently)

 

[Nothinman]

It's not recommended but I've done it and it worked fine, although on first resume it asks you about the VMs UUID and warns you about possible problems if the hardware is different.

 

[Brazen]

Originally posted by: AstroCreep
Here's what's funny - we're probably going to go to ESX within the next year (just a matter of time and planning). But we need something to use now (the IT battlecry, as it were). ...and unfortunately I have never used Linux in my life (and my current 'Push' on the education front is along the MCSA/E path).

Well you better get comfortable with linux before your ESX Server arrives. ESX Server is built on top of RedHat Enterprise Linux 3. A lot of things can be done through the gui, but one thing I always have to log in through ssh is to copy iso files of our installation media up to the server. Also, setting up ntp (if you want ntp, which you definately should) has to be done at the console and updating the ESX Server software is done at the console.

 

[AstroCreep]

I know ESX runs on top of a Linux kernel, but I guess I didn't realize that it would be more 'intense' than just following a few pre-configured scripts, selecting different options and what-not for setup (then administering it from the 'console' on a seperate server).

So I better get crackin' on the Linux book-learnin' huh?

 

[Pirotech]

Take a look at
Acronis True Image Echo Server for Windows

 

[AstroCreep]

Originally posted by: Pirotech
Take a look at
Acronis True Image Echo Server for Windows

Looks like a nice product; very similiar to Backup Exec System recovery (formerly 'Symantec Live State'; essentially the 'server' version of Ghost), but how would I run that, in the VMs?
Have you used this one before?

 

[yinan]

You can just continue to back them up over the network like you would a "normal" sever. Backup Exec, Netbackup dont know the difference between a physical machine vs a physical machine.

Dont make it harder than it has to be.

Jim

 

[Nothinman]

Except that I don't know of a single piece of Windows backup software that doesn't suck terribly, but you'd have that same problem with physical machines too.

 

[AstroCreep]

You can just continue to back them up over the network like you would a "normal" sever. Backup Exec, Netbackup dont know the difference between a physical machine vs a physical machine.

I'm still planning on normal 'File-Level' backups (w/ Backup Exec agents, which yes Nothinman, they do suck and I'm not a huge Veritas fan since the Symantec purchase), but we would like to ALSO be able to taake a backup of the VM from a standpoint where I can start the VM on a different piece of hardware, if it would ever be needed.

So yeah, I would have two backups every night.

 

[Brazen]

Originally posted by: AstroCreep
I know ESX runs on top of a Linux kernel, but I guess I didn't realize that it would be more 'intense' than just following a few pre-configured scripts, selecting different options and what-not for setup (then administering it from the 'console' on a seperate server).

So I better get crackin' on the Linux book-learnin' huh?

Well, it seemed pretty easy to me (as far as the text console part), but I was already well familiar with it. It couldn't hurt anyway.

 

[Genx87]

Almost a year later, EMC's tech support still has not gotten us to where we can restore a virtual machine from tape backup /rant.

This is my fear. I am evaluating ESX right now and the backup looks cumbersome to say the least. Has anybody played with Symantecs Linux\unix backup agent? One of the reasons I want to virtualize our infrastructure is for disaster recovery. In other words have the ability to take a backup of an entire virtual machine while it is running and if it craps, to pull it from a backup and have it running within mins. No rebuild of a virtual machine.

I am very surprised the marke leader has such a clumsy way to backup their virtual machines. Anybody played with Microsofts Hyper V? I want to but lack 64 bit hardware to play with the RC that Hyper V comes with.