Post by Bill LandryPost by Bill LandryPost by Dennis DavisDate: Mon, 05 Mar 2007 23:39:58 -0800
Subject: Re: [Clamav-users] msrbl sigs: rsync
...
# Check for MSRBL IMAGE database update
rsync -a rsync://rsync.mirror.msrbl.com/msrbl/MSRBL-Images.hdb \
$rsync_dir/MSRBL-Images.ndb
cp $rsync_dir/MSRBL-Images.ndb $tmp_dir
Shouldn't that read "MSRBL-Images.hdb" in the last two lines above?
Yes, you are correct, thanks for catching that (damn keyboard
viruses!) ;-)
Bill
Here is my latest script iteration, which now includes testing for newer
files before copying the file to the temp working directory for testing,
and when copying is done due to a newer file being found, the original
timestamps will be now preserved on the copied files.
I took just a quick look but it appears you are doing a time comparison
to a moved file, not the original file. Also - with just a teeny bit of
work you can reduce this to a single each curl and rsync invocation
rather than two each.
here's my version not that it's any great shakes, but it does work.
There's some installation steps to using it. You have to create the tmp
directory somewhere or use the example, and you have to create some
empty time tagging files (it uses a make-like paradigm). They are:
newspam, newphish, newscam, and newimages. These go in the tmp
directory. There are two test files that list the files to download. The
file names and contents are:
file.list
http://www.sanesecurity.com/clamav/phish.ndb.gz
http://www.sanesecurity.com/clamav/scam.ndb.gz
msrbl.list
MSRBL-Images.hdb
MSRBL-SPAM.ndb
Some of the complexity is a consequence of wget return codes being less
than helpful. It is the same whether a file is fetched or not.
Neither wget nor rsync will download a file unless the source is newer
than the local file. The post-fetch processing won't run unless the
local pattern files are newer than the time tagging files. It tries to
not waste time and cpu. The downloaded files are not modified in any way
so they retain their times and sizes. Rsync is used to put the
downloaded files into the working directory and this is an atomic
process so clamd doesn't barf. wget and rsync run once to get all four
files (or more if the vendors add to their list).
There is a 900 second randomizer so that this can run from cron but be a
bit agnostic of the cron cycle. The intent is to prevent my systems from
piling on the remote servers at regular intervals. Folks forget that
there are 60 minutes in the hour to set cron to run but so very many set
things to fire at 00 minutes. To get an immediate update enter any
string as an argument. If $1 is not empty the process will skip the
randomizer. There is a safety valve built in that prevents multiple
copies of this script from running. If an earlier instance is discovered
the new invocation will kill it and die. The cron cycle is such that it
should be only a broken instance that would be found still running and
this tries to clean things up.
And it is written for Solaris.
--------- 8< cut here ------------
#!/bin/bash
# usage: sanesecurity.sh [now]
# Arg "now" overrides random delay
RunFlag="/var/tmp/sane"
WorkingDirectory="/usr/local/share/clamav/tmp"
FileList="/usr/local/share/clamav/tmp/file.list"
MsrblServer="rsync://rsync.mirror.msrbl.com/msrbl/"
MsrblList="/usr/local/share/clamav/tmp/msrbl.list"
if [ -f "$RunFlag" ]; then
echo "This script already running. Cleaning up..."
/usr/bin/rm $RunFlag
/usr/bin/pkill sanesecurity.sh
else
/usr/bin/touch $RunFlag
fi
# sleep random 900 seconds to prevent cron lockstep
# with other clients. Use any command line arg to force
# immediate update. ARG[1] is arbitrary string.
if [ -z "$1" ]; then
sleep $[ RANDOM % 900 ]
fi
cd $WorkingDirectory
# Get Sane Security
/usr/local/bin/wget -q -N --input-file=$FileList >/dev/null 2>&1
# Process gzip files From SaneSecurity
if /usr/bin/test phish.ndb.gz -nt newphish; then
/usr/bin/gunzip < phish.ndb.gz > phish.ndb
/usr/local/bin/clamscan --quiet -d phish.ndb clam.txt && \
/usr/local/bin/rsync phish.ndb /usr/local/share/clamav || \
echo "phish.ndb is corrupt"
/usr/bin/settime -f phish.ndb.gz newphish
fi
if /usr/bin/test scam.ndb.gz -nt newscam; then
/usr/bin/gunzip < scam.ndb.gz > scam.ndb
/usr/local/bin/clamscan --quiet -d scam.ndb clam.txt && \
/usr/local/bin/rsync scam.ndb /usr/local/share/clamav || \
echo "scam.ndb is corrupt"
/usr/bin/settime -f scam.ndb.gz newscam
fi
# Get MSRBL files
/usr/local/bin/rsync -a --quiet --files-from=$MsrblList $MsrblServer
$WorkingDirectory >/dev/null 2>&1
# Processess text files from MSRBL
if /usr/bin/test MSRBL-Images.hdb -nt newimages; then
/usr/local/bin/clamscan --quiet -d MSRBL-Images.hdb clam.txt && \
/usr/local/bin/rsync MSRBL-Images.hdb /usr/local/share/clamav || \
echo "MSRBL-Images.hdb is corrupt"
/usr/bin/settime -f MSRBL-Images.hdb newimages
fi
if /usr/bin/test MSRBL-SPAM.ndb -nt newspam; then
/usr/local/bin/clamscan --quiet -d MSRBL-SPAM.ndb clam.txt && \
/usr/local/bin/rsync MSRBL-SPAM.ndb /usr/local/share/clamav || \
echo "MSRBL-SPAM.ndb is corrupt"
/usr/bin/settime -f MSRBL-SPAM.ndb newspam
fi
# clear run flag
/usr/bin/rm $RunFlag >/dev/null 2>&1
------------ >8 cut here -------------
dp
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://lurker.clamav.net/list/clamav-users.html