[clamav-users] A workaround for the major ClamAV DB update delays we have been experiencing

[Eric Tykwinski]

This is getting rather technical, and probably some of CloudFlare’s secret sauce.
It sounds like the anycast DNS that cloudflare hosts isn’t really working, or at least I would assume that they are using anycast.

So you query current.cvd.clamav.net <http://current.cvd.clamav.net/> but are getting different results at IAD and BOS. Now next is the inclusion of Comcast, which may and probably is caching DNS records beyond normal TTLs which could cause the difference. I personally always run an Unbound cache server on my mailserver networks to cache dns for at least an hour for rbls that I’m not rsyncing, but that could cause an issue with Microsoft’s wonderful 10 second MX records. So that’s where I’ve run into this issue, but not often enough since I’m just caching for an hour and probably MS expects it.

So my guess, is probably not anycast, but a caching DNS server that is still giving older records.

Sincerely,

Eric Tykwinski
TrueNet, Inc.
P: 610-429-8300

As some of you may be aware, ever since ClamAV began using Cloudflare,
we have seen many occasions when files like daily.cvd were not
available to our LAN until well after the DNS TXT record implied they
should be.
However, we discovered that these same files *are* available to our
Web/email server right away. So what is the difference? The first
difference is that our Web server (a VM) is offsite, and is served by
the "IAD" Cloudflare complex, whereas our local setup is served by the
"BOS" Cloudflare complex.
The second, and likely explanatory difference, is that our local setup
is connected via Comcast (a dynamic IP and all that), while our Web
server (with its static IP etc.) is almost certainly more directly
connected to the Internet as a whole.
The workaround we have adopted is as follows: we installed a "tinyproxy"
server on our offsite VM. To ensure it only proxys for us, it listens on
the encrypted OpenVPN tunnel we already had in place for FTP uploads
etc. Then, instead of directly accessing database.clamav.net, freshclam
uses our remote VM as a proxy,so that the cvd files are downloaded
indirectly from Cloudflare's IAD server complex (via tinyproxy) rather
than directly from Cloudflare's BOS server complex.
Since switching to this workaround a few days ago, we haven't observed
any delays: the cvd files are available right away when the DNS TXT
query says they should be.
I strongly suspect that Comcast is the culprit in the delays that had
plagued us. This is especially suggested by the fact that Cloudflare
Cache-Control: public, max-age=13672
where the max-age value varies, but is often several hours.
In my opinion, for data like ClamAV virus updates, the "Cache-Control:"
should specify "no-cache". Can Cloudflare do this for ClamAV?
---------------------------------------------------------------------
Below is a pair of recent (pre-workaround) log excerpts. They show a
delay of over 2.5 hours experienced from BOS (via Comcast) vs no delay
from IAD.
Note that the BOS "Date:" timestamp of 16:49:01 GMT *still* shows
a "Last-Modified:" timestamp of 06:15:18 GMT, while IAD already shows
the up-to-date "Last-Modified:" timestamp of 14:14:30 GMT at the much
earlier "Date:" of 14:29:01 GMT!
IAD
Date: Sun, 02 Dec 2018 14:09:01 GMT
Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
ClamAV-VDB:02 Dec 2018 01-14 -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
Date: Sun, 02 Dec 2018 14:29:01 GMT
Last-Modified: Sun, 02 Dec 2018 14:14:30 GMT
ClamAV-VDB:02 Dec 2018 09-13 -0500:25173:2167842:63:ba557f61737b9d4b66acc96f7044b524:3nBAOxo97ssSNZb
BOS
Date: Sun, 02 Dec 2018 14:09:01 GMT
Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
ClamAV-VDB:02 Dec 2018 01-14 -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
Date: Sun, 02 Dec 2018 14:29:01 GMT
Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
ClamAV-VDB:02 Dec 2018 01-14 -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
Date: Sun, 02 Dec 2018 14:49:01 GMT
Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
ClamAV-VDB:02 Dec 2018 01-14 -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
Date: Sun, 02 Dec 2018 15:09:01 GMT
Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
ClamAV-VDB:02 Dec 2018 01-14 -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
Date: Sun, 02 Dec 2018 15:29:02 GMT
Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
ClamAV-VDB:02 Dec 2018 01-14 -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
Date: Sun, 02 Dec 2018 15:49:02 GMT
Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
ClamAV-VDB:02 Dec 2018 01-14 -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
Date: Sun, 02 Dec 2018 16:09:01 GMT
Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
ClamAV-VDB:02 Dec 2018 01-14 -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
Date: Sun, 02 Dec 2018 16:29:01 GMT
Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
ClamAV-VDB:02 Dec 2018 01-14 -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
Date: Sun, 02 Dec 2018 16:49:01 GMT
Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
ClamAV-VDB:02 Dec 2018 01-14 -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
Date: Sun, 02 Dec 2018 17:09:01 GMT
Last-Modified: Sun, 02 Dec 2018 14:14:30 GMT
ClamAV-VDB:02 Dec 2018 09-13 -0500:25173:2167842:63:ba557f61737b9d4b66acc96f7044b524:3nBAOxo97ssSNZb
_______________________________________________
clamav-users mailing list
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
https://github.com/vrtadmin/clamav-faq
http://www.clamav.net/contact.html#ml

[Eric Tykwinski]

Paul,

Sorry I got it backwards, I thought you were saying the TXT record was different which would be effected by DNS caching.
The CloudFlare cache would definitely effect daily.cvd, but updates are new.

Only way I could see you get around it yourself is to create your own cdiff program from the source and use the updates,
which pretty much is using freshclam.

Sincerely,

Eric Tykwinski
TrueNet, Inc.
P: 610-429-8300

Not sure what DNS caching would have to do with this. As I understand
"anycast", it happens at the IP address level. An anycast IP address
gets routed differently depending are where you are -- different
(regional) routers have different "next hops" for the IP address, and
it eventually ends up at a "nearby" server. This is in addition to the
fact that database.clamav.net resolves to 5 different IP addresses (all
of which are anycast IPs, I would guess).
The Cloudflare servers, although they have the same IP address(es) seem
CF-RAY: 4852e02c35ae5a5c-BOS
CF-RAY: 48526af5624356c3-IAD
Finally, AFAIK, I always get the same result for
"dig TXT current.cvd.clamav.net"
no matter where I 'dig' (or 'host') from.
On Fri, 7 Dec 2018 19:27:20 -0500

Post by Eric Tykwinski
This is getting rather technical, and probably some of CloudFlare’s
secret sauce. It sounds like the anycast DNS that cloudflare hosts
isn’t really working, or at least I would assume that they are using
anycast.
So you query current.cvd.clamav.net <http://current.cvd.clamav.net/>
but are getting different results at IAD and BOS. Now next is the
inclusion of Comcast, which may and probably is caching DNS records
beyond normal TTLs which could cause the difference. I personally
always run an Unbound cache server on my mailserver networks to cache
dns for at least an hour for rbls that I’m not rsyncing, but that
could cause an issue with Microsoft’s wonderful 10 second MX
records. So that’s where I’ve run into this issue, but not often
enough since I’m just caching for an hour and probably MS expects it.
So my guess, is probably not anycast, but a caching DNS server that
is still giving older records.
Sincerely,
Eric Tykwinski
TrueNet, Inc.
P: 610-429-8300

As some of you may be aware, ever since ClamAV began using
Cloudflare, we have seen many occasions when files like daily.cvd
were not available to our LAN until well after the DNS TXT record
implied they should be.
However, we discovered that these same files *are* available to our
Web/email server right away. So what is the difference? The first
difference is that our Web server (a VM) is offsite, and is served
by the "IAD" Cloudflare complex, whereas our local setup is served
by the "BOS" Cloudflare complex.
The second, and likely explanatory difference, is that our local
setup is connected via Comcast (a dynamic IP and all that), while
our Web server (with its static IP etc.) is almost certainly more
directly connected to the Internet as a whole.
The workaround we have adopted is as follows: we installed a
"tinyproxy" server on our offsite VM. To ensure it only proxys for
us, it listens on the encrypted OpenVPN tunnel we already had in
place for FTP uploads etc. Then, instead of directly accessing
database.clamav.net, freshclam uses our remote VM as a proxy,so
that the cvd files are downloaded indirectly from Cloudflare's IAD
server complex (via tinyproxy) rather than directly from
Cloudflare's BOS server complex.
Since switching to this workaround a few days ago, we haven't
observed any delays: the cvd files are available right away when
the DNS TXT query says they should be.
I strongly suspect that Comcast is the culprit in the delays that
had plagued us. This is especially suggested by the fact that
Cache-Control: public, max-age=13672
where the max-age value varies, but is often several hours.
In my opinion, for data like ClamAV virus updates, the
"Cache-Control:" should specify "no-cache". Can Cloudflare do this
for ClamAV?
---------------------------------------------------------------------
Below is a pair of recent (pre-workaround) log excerpts. They show a
delay of over 2.5 hours experienced from BOS (via Comcast) vs no
delay from IAD.
Note that the BOS "Date:" timestamp of 16:49:01 GMT *still* shows
a "Last-Modified:" timestamp of 06:15:18 GMT, while IAD already
shows the up-to-date "Last-Modified:" timestamp of 14:14:30 GMT at
the much earlier "Date:" of 14:29:01 GMT!
IAD
Date: Sun, 02 Dec 2018 14:09:01 GMT
Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
ClamAV-VDB:02 Dec 2018 01-14
-0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
Date: Sun, 02 Dec 2018 14:29:01 GMT
Last-Modified: Sun, 02 Dec 2018 14:14:30 GMT
ClamAV-VDB:02 Dec 2018 09-13
-0500:25173:2167842:63:ba557f61737b9d4b66acc96f7044b524:3nBAOxo97ssSNZb
BOS
Date: Sun, 02 Dec 2018 14:09:01 GMT
Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
ClamAV-VDB:02 Dec 2018 01-14
-0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
Date: Sun, 02 Dec 2018 14:29:01 GMT
Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
ClamAV-VDB:02 Dec 2018 01-14
-0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
Date: Sun, 02 Dec 2018 14:49:01 GMT
Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
ClamAV-VDB:02 Dec 2018 01-14
-0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
Date: Sun, 02 Dec 2018 15:09:01 GMT
Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
ClamAV-VDB:02 Dec 2018 01-14
-0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
Date: Sun, 02 Dec 2018 15:29:02 GMT
Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
ClamAV-VDB:02 Dec 2018 01-14
-0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
Date: Sun, 02 Dec 2018 15:49:02 GMT
Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
ClamAV-VDB:02 Dec 2018 01-14
-0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
Date: Sun, 02 Dec 2018 16:09:01 GMT
Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
ClamAV-VDB:02 Dec 2018 01-14
-0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
Date: Sun, 02 Dec 2018 16:29:01 GMT
Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
ClamAV-VDB:02 Dec 2018 01-14
-0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
Date: Sun, 02 Dec 2018 16:49:01 GMT
Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
ClamAV-VDB:02 Dec 2018 01-14
-0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT
Date: Sun, 02 Dec 2018 17:09:01 GMT
Last-Modified: Sun, 02 Dec 2018 14:14:30 GMT
ClamAV-VDB:02 Dec 2018 09-13
-0500:25173:2167842:63:ba557f61737b9d4b66acc96f7044b524:3nBAOxo97ssSNZb

_______________________________________________
clamav-users mailing list
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
https://github.com/vrtadmin/clamav-faq
http://www.clamav.net/contact.html#ml

[Eric Tykwinski]

J.R.

You are falling into the same trap I followed. The txt record is:
current.cvd.clamav.net. 1749 IN TXT "0.101.0:58:25189:1544315340:1:63:48210:327"

But host headers is what he’s looking at:
telnet database.clamav.net 80
Trying 104.16.185.138...
Connected to database.clamav.net.cdn.cloudflare.net.
Escape character is '^]'.
GET /daily.cvd HTTP/1.1
host: database.clamav.net

HTTP/1.1 200 OK
Date: Sun, 09 Dec 2018 01:18:51 GMT
Content-Type: application/octet-stream
Content-Length: 53110330
Connection: keep-alive
Set-Cookie: __cfduid=ddc4d2ab2a13638c99a90bb14c12128971544318331; expires=Mon, 09-Dec-19 01:18:51 GMT; path=/; domain=.clamav.net; HttpOnly
Last-Modified: Sat, 08 Dec 2018 18:18:00 GMT
ETag: "5c0c0ad8-32a663a"
Expires: Sun, 09 Dec 2018 05:05:51 GMT
Cache-Control: public, max-age=13620
CF-Cache-Status: HIT
Accept-Ranges: bytes
Server: cloudflare
CF-RAY: 4863a3a9553bc5d2-EWR

ClamAV-VDB:08 Dec 2018 13-18 -0500:25189:2177974:63:2e2e28a4556e83e2df68c40fa61566d4:nWqDCF65xA9fMhiKYOtZhH8Up6lAHLrl6VyCrXRAXCB7aMf7WqSPrwMz/YHhdgKSNjxGiL8Z2ORQ2aPm23KwqwyJUpOZv94+soWx+NibPlKBPJ6/ZAt9Z5UrhgDbgz0IVQsHX998ZjBE6NY6xtqfzboOPNKyeFINLeAUL5hSpzj:neo:1544293134

So daily.cvd is being cached on cloudflare for the first update and you might need to be running a freshclam right after a new install since it’s out of date due to caching on cloudflare’s server.

Sincerely,

Eric Tykwinski
TrueNet, Inc.
P: 610-429-8300

I've kind of been reading this thread about the delay at one location
vs the other.
Maybe I missed it, but I don't seem to recall which DNS servers you
were querying. I remember you saying the one location you were having
the issues was Comcast as the ISP, but were you always using the
Comcast DNS or did you try others like 1.1.1.1 or 8.8.8.8 ?
Or was the DNS saying there was a newer version but when you queried
cloudflare it was reporting differently?
_______________________________________________
clamav-users mailing list
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
https://github.com/vrtadmin/clamav-faq
http://www.clamav.net/contact.html#ml

_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clama

[Eric Tykwinski]

Joel,

Not sure what you’re saying here. Are you saying that the daily on the cache is out of date?

I haven’t really noticed it, but that was Paul Kosinski’s observation from what I’m reading in the first email.
So it looks like IAD updated at 14:14:30 GMT, but BOS didn’t update till 17:09:01 GMT from his email.

From back in archives, I think he’s using wget to just pull the files, but freshclam would just pull the cdiffs and keep you up to date on the next check.

Sincerely,

Eric Tykwinski
TrueNet, Inc.
P: 610-429-8300

[Joel Esler (jesler)]

As it should be. No one should be downloading the daily and main, (although thousands are), cdiffs were created for a reason.

Sent from my  iPhone

Post by Eric Tykwinski
From back in archives, I think he’s using wget to just pull the files, but freshclam would just pull the cdiffs and keep you up to date on the next check.

[Dennis Peterson]

Exactly right. We can't be blaming the ClamAV process when we don't use the
ClamAV process. People that don't use freshclam should have no expectation of
high reliability. In fact any expectations are baseless when the wrong tools are
employed.

dp

Post by Joel Esler (jesler)
As it should be. No one should be downloading the daily and main, (although thousands are), cdiffs were created for a reason.
Sent from my  iPhone

Post by Eric Tykwinski
From back in archives, I think he’s using wget to just pull the files, but freshclam would just pull the cdiffs and keep you up to date on the next check.
_______________________________________________
clamav-users mailing list
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
https://github.com/vrtadmin/clamav-faq
http://www.clamav.net/contact.html#ml

[Gary R. Schmidt]

Post by Dennis Peterson
Exactly right. We can't be blaming the ClamAV process when we don't use
the ClamAV process. People that don't use freshclam should have no
expectation of high reliability. In fact any expectations are baseless
when the wrong tools are employed.

Sigh.

Does no one actually READ THE MESSAGES???

The OP's problem is:

FRESHCLAM FAILS, REPEATEDLY, UNTIL ALL MIRRORS ARE MARKED AS BAD
AND NO UPDATES CAN OCCUR.

Pissing up a rope about "you shouldn't do various work-arounds" is a
waste of time and bandwidth.

The OP has shown that different Cloudflare nodes give (him) different
results, someone should be asking CLoudflare about how this can be
addressed, not dismissing the very valid and basic problem.

This sort of behaviour just proves that Dunning-Kruger is alive and
involved in far too many OSS projects.

Cheers,
Gary B-)
_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[Dennis Peterson]

Helps too to read the entire thread and the thread that preceded this one. The
OP has used combinations of dig and wget in diagnosing his problems.

dp

Post by Gary R. Schmidt

Post by Dennis Peterson
Exactly right. We can't be blaming the ClamAV process when we don't use the
ClamAV process. People that don't use freshclam should have no expectation of
high reliability. In fact any expectations are baseless when the wrong tools
are employed.

Sigh.
Does no one actually READ THE MESSAGES???
    FRESHCLAM FAILS, REPEATEDLY, UNTIL ALL MIRRORS ARE MARKED AS BAD
    AND NO UPDATES CAN OCCUR.
Pissing up a rope about "you shouldn't do various work-arounds" is a waste of
time and bandwidth.
The OP has shown that different Cloudflare nodes give (him) different results,
someone should be asking CLoudflare about how this can be addressed, not
dismissing the very valid and basic problem.
This sort of behaviour just proves that Dunning-Kruger is alive and involved
in far too many OSS projects.
    Cheers,
        Gary    B-)
_______________________________________________
clamav-users mailing list
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
https://github.com/vrtadmin/clamav-faq
http://www.clamav.net/contact.html#ml

_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/conta

[Eric Tykwinski]

Dennis,

Helps too to read the entire thread and the thread that preceded this one. The OP has used combinations of dig and wget in diagnosing his problems.
dp

Seriously, then he should be just trying to pull the new cdiffs to see if they are propagated to the various Cloudflare hosts.

Post by Gary R. Schmidt
Sigh.
Does no one actually READ THE MESSAGES???
FRESHCLAM FAILS, REPEATEDLY, UNTIL ALL MIRRORS ARE MARKED AS BAD
AND NO UPDATES CAN OCCUR.
Pissing up a rope about "you shouldn't do various work-arounds" is a waste of time and bandwidth.
The OP has shown that different Cloudflare nodes give (him) different results, someone should be asking CLoudflare about how this can be addressed, not dismissing the very valid and basic problem.
This sort of behaviour just proves that Dunning-Kruger is alive and involved in far too many OSS projects.
Cheers,
Gary B-)

Gary,

I haven’t really followed the whole thread, but I’ve been seeing it for months that I recall, definitely a waste of bandwidth, and probably should be solved to some extent.

Looking at his logs, the headers are only for a CVD, so he’s not trying updates.

Example of a cdiff pull from telnet:
telnet database.clamav.net 80
Trying 104.16.186.138...
Connected to database.clamav.net.cdn.cloudflare.net.
Escape character is '^]'.
GET /daily-25195.cdiff HTTP/1.1
host: database.clamav.net

?????o??_}??/~?uЯ?|??~?f?l??Ox????????~??????O6????/??????_?????>??Ϸ_????7?~??̯???ߢ?????ӏ~???B??{}~?[????A???7????ņ?>???


You don’t get those nice header parts to the file, so you wouldn’t know the last update as it’s apart of the file itself. Looking at manager.c on freshclam, he should have been posting something like: "^getfile: %s not found on %s (IP: %s)\n" which gets posted to the logs when the file doesn’t exist.

I’m not positive on this so Micah can chime in, but I do believe you get the cdiff files from the DNS TXT somehow.

If anything it’s a good lesson on how exactly freshclam works.

Sincerely,

Eric Tykwinski