rsync hanging at "building file list"

[Red Squirrel]

I have this script I've used for a long time, basically it's just to sync projects from dev to test and test to prod so I can work on it locally then commit changes to test for further testing and then to prod. For some reason, suddently, it started not working. It just hangs at "building file list" for the rsync command.

The rsync command being used is this:

rsync --checksum --delete-after -rva --exclude-from="${maindir}${envfr}/${proj}/syncexcludes_src.txt" "${maindir}${envfr}/${proj}/src/" "${maindir}${env}/${proj}/src/"

This worked fine like a week ago, it just randomly started doing this. I wanted to make a change to a website but it would not sync so had to SSH to the prod server and do the change manually. Any clues as to what could cause this?

This is the full script:

#!/bin/sh

maindir="/network/appdev/"

env=$1
envfr=""

if [ "$env" == "test" ] ; then
echo "syncing to test"
envfr="dev"
elif [ "$env" == "prod" ] ; then
echo "syncing to prod"
envfr="test"
else
echo "specify test or prod (environment to sync TO)"
echo "Usage: ./sync.sh [test/prod] [projectname]"
exit
fi

proj=$2

if [ "$proj" == "" ] ; then
echo "You must specify a project folder (and it has to have a src folder in it)"
exit
fi

if [ ! -d "${maindir}${envfr}/${proj}/src/" ] ; then
echo "Project ${proj} does not exist in ${envfr} or src folder not present"
exit
fi

#    File change reason flags
#
#    c -> checksum differs
#    s -> size differs
#    t -> mtime differs
#    p -> permissions differ
#    o -> owners
#    g -> group
#    u -> reserved for future use
#    a -> ACLs
#    x -> extended attributes
#    + -> is new (all +++++'s indicates a new file)
#    . -> is unchanged


echo "checks ok, syncing..."


echo "sync: src"
touch ${maindir}${envfr}/${proj}/syncexcludes_src.txt
mkdir -p "${maindir}${env}/${proj}/src/"

echo
echo "dir of destination:"
dir -ls "${maindir}${env}/${proj}/src/"
echo
echo "-------------------"

echo
echo "--------------------------------------------------------------------------------------------------------------------"
rsync --checksum --delete-after -rva --exclude-from="${maindir}${envfr}/${proj}/syncexcludes_src.txt" "${maindir}${envfr}/${proj}/src/" "${maindir}${env}/${proj}/src/"
echo "--------------------------------------------------------------------------------------------------------------------"
echo
echo "the following files were excluded as per syncexcludes_src.txt:"
echo
cat ${maindir}${envfr}/${proj}/syncexcludes_src.txt
echo


if [ -d "${maindir}${envfr}/${proj}/virdir/" ] ; then
echo "sync: virdirs"
touch ${maindir}${envfr}/${proj}/syncexcludes_virdir.txt
mkdir -p "${maindir}${env}/${proj}/virdir/"
echo
echo "---------------------------------------------------------------------------------------------------------------------"
rsync --checksum --delete-after -rva --exclude-from="${maindir}${envfr}/${proj}/syncexcludes_virdir.txt" "${maindir}${envfr}/${proj}/virdir/" "${maindir}${env}/${proj}/virdir/"
echo "---------------------------------------------------------------------------------------------------------------------"
echo
echo "the following files were excluded as per syncexcludes_virdir.txt:"
echo
cat ${maindir}${envfr}/${proj}/syncexcludes_virdir.txt
echo
echo "----"
fi



echo "Checking for secondary sync file..."
echo

cd ${maindir}${envfr}/${proj}

if [ -f "sync.sh" ] ; then
echo "executing ${proj}'s sync file..."
chmod +x sync.sh
./sync.sh
fi
echo
echo Completed.

 

[Fallen Kell]

Could be some form of file or inode corruption that could have occurred. If you get a chance, I would unmount both the source and destination filesystems and do a fsck on both of them. The only other suggestion is to run a trace on the rsync to see if you get more information about when it hangs. Something like "strace -f -ttt your-script-goes-here".

 

[Red Squirrel]

Hmm funny you mention that since today I was organizing some old files and noticed lot of them were randomly corrupted. I wonder if I have some silent corruption going on. Every now and then my whole network will just grind to a halt and I'll get a lot of kernel errors like "the task delayed for 120 seconds" on my VMs. Switching everything to async mode fixed that but I wonder if I'm getting corruption now instead of lockups. Running mdadm raid which has always been pretty solid but wonder if I should look into ZFS as it has better bit rot protection and other features to avoid corruption.

I need to find something other than NFS for presenting files though, I find I have serious performance issues with NFS and I pretty much gave up trying to figure out why. Setting everything to async did somewhat fix it, but that has a high risk of corruption. Wonder if that's what I'm seeing.