rsync wildcards

[zimu]

hey guys,

i'm trying to break up my rsync a bit so it handles one letter of the alphabet at a time.

so, rsync /home/a*, rsync /home/b* and so forth. however it seems to only copy the first folder with "a" and ignore all the rest.

i've tried rsyc /home but it pretty much does not work given how many folders and files i have - it spent 15 minutes just calculating the number of files which is a waste of time...


any ideas how to break up the task?

 

[Nothinman]

Counting them isn't really a waste of time since it has to compare file lists and checksum all of the files. And the glob should be expanded by the shell and not rsync, so when you type 'rsync /home/a*' what the shell actually runs is 'rsync /home/a1 /home/a2 etc'.

 

[zimu]

hi thanks for the response!

re the last bit of what you wrote, that's exactly wha ti want it to do! e.g. if i have:
/home/aa
/home/ab
/home/ac
/home/ba
/home/bb

and i want to rsync all the a's (aa,ab,ac) then i want to type rsync [options] a*.
however doing that only rsyncs aa, not ab/ac.
how do i rsync only aa,ab,ac out of the list above?

thanks!

 

[Nothinman]

It should be doing it already unless the shell isn't expanding /home/a* properly for some reason. Does 'echo /home/a*' give you the right list?

 

[zimu]

the /home/a* shows all the folders i want - i.e. aa/ab/ac.
this is messed up!

the command i'm running is:
rsync --links --stats --progress --verbose --backup --recursive /home/a* rsyncserver::location

 

[Nothinman]

Why not just let it run over all of /home in one run just once and see how long it takes, I doubt you'll save any time by breaking up. If anything it'll probably slow it down.

 

[zimu]

i tried that! quite a number of issues:
- 30gb in /home directory. i'd rather split it up by day
- i ran it last night (non verbose) and it ran the entire night. the next day it had frozen and had only transferred 300mb
- i ran it this afternoon (verbose). it counted 1,000,000+ files before it froze, even before transfer began. this took about 20 mins.

 

[Nothinman]

Are you sure it was frozen? I've never tried to rsync a million files but if it's really freezing instead of crashing or quitting with an error I'd suspect some problems somewhere else.

 

[skyking]

Are you using rsync primarily for backup purposes?
If so, take a look at dirvish.
It uses rsync to make incremental backups, and may have some features that are native, that you are trying to script.

 

[n0cmonkey]

Seems a bit overboard, but this might work. I guess I don't have enough files, I've only transfered ~60GB of bigger files using rsync.