[clamav-users] Latest report on update "delays"

[Micah Snyder (micasnyd)]

Hi Paul,

Thanks for the update.

I am interested to know how freshclam in ClamAV 0.100.2 performs for you. I have made some tweaks to make it ignore mirrors for less time, but more importantly I implemented a change to have it report "up to date" in the event that the signature version provided by the mirror is 1 behind what was advertised. My hope is that this alleviates the issue.

Respectfully,
Micah


Micah Snyder
ClamAV Development
Talos
Cisco Systems, Inc.


On Oct 4, 2018, at 4:47 PM, Paul Kosinski <clamav-***@iment.com<mailto:clamav-***@iment.com>> wrote:

At Joel's suggestion, i have changed our sampling rate looking for ClamAV cvd updates from 15 minutes down to 1 minute. This gives a more precise measurement of how long it takes for the cvd file(s) to actually become available from Cloudflare after its presence is "advertised" by the CNS TXT record.

Since these measurements are mainly useful for tuning the ClamAV servers, I won't in the future post them to clamav-users unless others besides the ClamAV team find them useful. (Maybe they should go to the clamav-developers list?)

In any case, here is the latest log of delays. Note that these more precisely measured delays are not explained as mere 15-minute quantization errors.

2018-10-02 09:18:02 No delay
2018-10-02 17:18:02 No delay
2018-10-03 01:31:02 00:13:00 delay
2018-10-03 09:42:02 00:24:00 delay
2018-10-03 17:52:02 00:33:59 delay
2018-10-04 01:18:02 No delay
2018-10-04 09:40:01 00:21:59 delay
_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net<mailto:clamav-***@lists.clamav.net>
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[Micah Snyder (micasnyd)]

Hi Paul,

I realize it may look misleading to state that you're up to date when a newer database has been announced. However, if the newer database is still being uploaded to the CDN, it is more accurate to say that the DNS announcement is premature.

The change to freshclam is an effort to ignore potentially premature database version numbers listed via DNS.

Micah Snyder
ClamAV Development
Talos
Cisco Systems, Inc.


On Oct 15, 2018, at 2:26 PM, Paul Kosinski <clamav-***@iment.com<mailto:clamav-***@iment.com>> wrote:

I don't have time at the present to try out 0.100.2. I am rebuilding
our Web server, which had a disk crash. We have backups, but we need
whole new hardware since the old server had an old 32-bit-only CPU.
Thus a *supported* Linux version will not run, and so a simple disk
replacement was not a viable option. (Unfortunately the new server,
although only a VM, still costs almost 50% more per month than the old
raw hardware, which was adequate, if clunky.)

Back to ClamAV: I don't much like the idea of saying signatures are "up
to date" if only 1 version behind the latest version. Most of the time
that won't matter, but sometimes a really urgent new signature comes
out and this approach could mislead people into a false sense of
security.



On Thu, 4 Oct 2018 22:27:14 +0000
"Micah Snyder (micasnyd)" <***@cisco.com<mailto:***@cisco.com>> wrote:

Hi Paul,

Thanks for the update.

I am interested to know how freshclam in ClamAV 0.100.2 performs for
you. I have made some tweaks to make it ignore mirrors for less
time, but more importantly I implemented a change to have it report
"up to date" in the event that the signature version provided by the
mirror is 1 behind what was advertised. My hope is that this
alleviates the issue.

Respectfully,
Micah


Micah Snyder
ClamAV Development
Talos
Cisco Systems, Inc.


On Oct 4, 2018, at 4:47 PM, Paul Kosinski
<clamav-***@iment.com<mailto:clamav-***@iment.com><mailto:clamav-***@iment.com>> wrote:

At Joel's suggestion, i have changed our sampling rate looking for
ClamAV cvd updates from 15 minutes down to 1 minute. This gives a
more precise measurement of how long it takes for the cvd file(s) to
actually become available from Cloudflare after its presence is
"advertised" by the CNS TXT record.

Since these measurements are mainly useful for tuning the ClamAV
servers, I won't in the future post them to clamav-users unless
others besides the ClamAV team find them useful. (Maybe they should
go to the clamav-developers list?)

In any case, here is the latest log of delays. Note that these more
precisely measured delays are not explained as mere 15-minute
quantization errors.

2018-10-02 09:18:02 No delay
2018-10-02 17:18:02 No delay
2018-10-03 01:31:02 00:13:00 delay
2018-10-03 09:42:02 00:24:00 delay
2018-10-03 17:52:02 00:33:59 delay
2018-10-04 01:18:02 No delay
2018-10-04 09:40:01 00:21:59 delay
_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net<mailto:clamav-***@lists.clamav.net><mailto:clamav-***@lists.clamav.net>
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net<mailto:clamav-***@lists.clamav.net>
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[Joel Esler (jesler)]

The DNS announcement is made as the last step in the process. The lag that may be seen is the lag in between when the DNS update is posted, and before the file is pushed out to the Tier 1 CDN servers. It has to be requested at the CDN server before it is cached.



On Oct 18, 2018, at 12:07 PM, Micah Snyder (micasnyd) <***@cisco.com<mailto:***@cisco.com>> wrote:

Hi Paul,

I realize it may look misleading to state that you're up to date when a newer database has been announced. However, if the newer database is still being uploaded to the CDN, it is more accurate to say that the DNS announcement is premature.

The change to freshclam is an effort to ignore potentially premature database version numbers listed via DNS.

Micah Snyder
ClamAV Development
Talos
Cisco Systems, Inc.


On Oct 15, 2018, at 2:26 PM, Paul Kosinski <clamav-***@iment.com<mailto:clamav-***@iment.com>> wrote:

I don't have time at the present to try out 0.100.2. I am rebuilding
our Web server, which had a disk crash. We have backups, but we need
whole new hardware since the old server had an old 32-bit-only CPU.
Thus a *supported* Linux version will not run, and so a simple disk
replacement was not a viable option. (Unfortunately the new server,
although only a VM, still costs almost 50% more per month than the old
raw hardware, which was adequate, if clunky.)

Back to ClamAV: I don't much like the idea of saying signatures are "up
to date" if only 1 version behind the latest version. Most of the time
that won't matter, but sometimes a really urgent new signature comes
out and this approach could mislead people into a false sense of
security.



On Thu, 4 Oct 2018 22:27:14 +0000
"Micah Snyder (micasnyd)" <***@cisco.com<mailto:***@cisco.com>> wrote:

Hi Paul,

Thanks for the update.

I am interested to know how freshclam in ClamAV 0.100.2 performs for
you. I have made some tweaks to make it ignore mirrors for less
time, but more importantly I implemented a change to have it report
"up to date" in the event that the signature version provided by the
mirror is 1 behind what was advertised. My hope is that this
alleviates the issue.

Respectfully,
Micah


Micah Snyder
ClamAV Development
Talos
Cisco Systems, Inc.


On Oct 4, 2018, at 4:47 PM, Paul Kosinski
<clamav-***@iment.com<mailto:clamav-***@iment.com><mailto:clamav-***@iment.com>> wrote:

At Joel's suggestion, i have changed our sampling rate looking for
ClamAV cvd updates from 15 minutes down to 1 minute. This gives a
more precise measurement of how long it takes for the cvd file(s) to
actually become available from Cloudflare after its presence is
"advertised" by the CNS TXT record.

Since these measurements are mainly useful for tuning the ClamAV
servers, I won't in the future post them to clamav-users unless
others besides the ClamAV team find them useful. (Maybe they should
go to the clamav-developers list?)

In any case, here is the latest log of delays. Note that these more
precisely measured delays are not explained as mere 15-minute
quantization errors.

2018-10-02 09:18:02 No delay
2018-10-02 17:18:02 No delay
2018-10-03 01:31:02 00:13:00 delay
2018-10-03 09:42:02 00:24:00 delay
2018-10-03 17:52:02 00:33:59 delay
2018-10-04 01:18:02 No delay
2018-10-04 09:40:01 00:21:59 delay
_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net<mailto:clamav-***@lists.clamav.net><mailto:clamav-***@lists.clamav.net>
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net<mailto:clamav-***@lists.clamav.net>
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net<mailto:clamav-***@lists.clamav.net>
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[Eric Tykwinski]

As far as I know you don't upload to cloudflare, it's more of how often does
cloudflare check to see if the files have changed.
So you setup a TTL on the check frequency on the cloudflare website.

Since updates are new they should just be pulled when you ask from the main
clam server.
So you ask for daily-25048.cdiff, and Cloudflare will ask Clam's main server
for that file and cache it.

So my guess would be same as the TTL on the DNS check:
current.cvd.clamav.net. 1800 IN TXT
"0.100.2:58:25048:1539883740:1:63:48006:327"
I.E. 30 minutes for older files, and new ones are when they come in.

Sound about right Joel, Micah?

Sincerely,

Eric Tykwinski
TrueNet, Inc.
P: 610-429-8300

> -----Original Message-----
> From: clamav-users [mailto:clamav-users-***@lists.clamav.net] On
> Behalf Of Paul Kosinski
> Sent: Thursday, October 18, 2018 1:23 PM
> To: clamav-***@lists.clamav.net
> Subject: Re: [clamav-users] Latest report on update "delays"
>
> How can it take 10, 20 30 or more minutes (and I've seen well over an
> hour at times) to upload the ClamAV database to Cloudflare? Does it have
> to be uploaded separately (and maybe sequentially) from Cisco to each
> Cloudflare mirror? Or is Cloudflare's automatic propagation slow?
>
>
> On Thu, 18 Oct 2018 16:07:38 +0000
> "Micah Snyder (micasnyd)" <***@cisco.com> wrote:
>
> > Hi Paul,
> >
> > I realize it may look misleading to state that you're up to date when
> > a newer database has been announced. However, if the newer database
> > is still being uploaded to the CDN, it is more accurate to say that
> > the DNS announcement is premature.
> >
> > The change to freshclam is an effort to ignore potentially premature
> > database version numbers listed via DNS.
> >
> > Micah Snyder
> > ClamAV Development
> > Talos
> > Cisco Systems, Inc.
> >
> >
> > On Oct 15, 2018, at 2:26 PM, Paul Kosinski
> > <clamav-***@iment.com<mailto:clamav-***@iment.com>> wrote:
> >
> > I don't have time at the present to try out 0.100.2. I am rebuilding
> > our Web server, which had a disk crash. We have backups, but we need
> > whole new hardware since the old server had an old 32-bit-only CPU.
> > Thus a *supported* Linux version will not run, and so a simple disk
> > replacement was not a viable option. (Unfortunately the new server,
> > although only a VM, still costs almost 50% more per month than the old
> > raw hardware, which was adequate, if clunky.)
> >
> > Back to ClamAV: I don't much like the idea of saying signatures are
> > "up to date" if only 1 version behind the latest version. Most of the
> > time that won't matter, but sometimes a really urgent new signature
> > comes out and this approach could mislead people into a false sense of
> > security.
> >
> >
> >
> > On Thu, 4 Oct 2018 22:27:14 +0000
> > "Micah Snyder (micasnyd)"
> > <***@cisco.com<mailto:***@cisco.com>> wrote:
> >
> > Hi Paul,
> >
> > Thanks for the update.
> >
> > I am interested to know how freshclam in ClamAV 0.100.2 performs for
> > you. I have made some tweaks to make it ignore mirrors for less
> > time, but more importantly I implemented a change to have it report
> > "up to date" in the event that the signature version provided by the
> > mirror is 1 behind what was advertised. My hope is that this
> > alleviates the issue.
> >
> > Respectfully,
> > Micah
> >
> >
> > Micah Snyder
> > ClamAV Development
> > Talos
> > Cisco Systems, Inc.
> >
> >
> > On Oct 4, 2018, at 4:47 PM, Paul Kosinski
> > <clamav-***@iment.com<mailto:clamav-
> ***@iment.com><mailto:clamav-***@iment.com>>
> > wrote:
> >
> > At Joel's suggestion, i have changed our sampling rate looking for
> > ClamAV cvd updates from 15 minutes down to 1 minute. This gives a
> > more precise measurement of how long it takes for the cvd file(s) to
> > actually become available from Cloudflare after its presence is
> > "advertised" by the CNS TXT record.
> >
> > Since these measurements are mainly useful for tuning the ClamAV
> > servers, I won't in the future post them to clamav-users unless
> > others besides the ClamAV team find them useful. (Maybe they should
> > go to the clamav-developers list?)
> >
> > In any case, here is the latest log of delays. Note that these more
> > precisely measured delays are not explained as mere 15-minute
> > quantization errors.
> >
> > 2018-10-02 09:18:02 No delay
> > 2018-10-02 17:18:02 No delay
> > 2018-10-03 01:31:02 00:13:00 delay
> > 2018-10-03 09:42:02 00:24:00 delay
> > 2018-10-03 17:52:02 00:33:59 delay
> > 2018-10-04 01:18:02 No delay
> > 2018-10-04 09:40:01 00:21:59 delay
> > _______________________________________________
> > clamav-users mailing list
> > clamav-***@lists.clamav.net<mailto:clamav-
> ***@lists.clamav.net><mailto:clamav-***@lists.clamav.net>
> > http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
>
> _______________________________________________
> clamav-users mailing list
> clamav-***@lists.clamav.net
> http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml


_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[Joel Esler (jesler)]

Cloudflare will grab the file from our infrastructure once it's been requested. (Otherwise it wouldn't know it was there, we can't push into Cloudflare.). But we have discussed a few ideas internally that I think will fix this, let us try a couple things and see if it cuts down on this.

On Oct 18, 2018, at 1:55 PM, Eric Tykwinski <eric-***@truenet.com<mailto:eric-***@truenet.com>> wrote:

As far as I know you don't upload to cloudflare, it's more of how often does
cloudflare check to see if the files have changed.
So you setup a TTL on the check frequency on the cloudflare website.

Since updates are new they should just be pulled when you ask from the main
clam server.
So you ask for daily-25048.cdiff, and Cloudflare will ask Clam's main server
for that file and cache it.

So my guess would be same as the TTL on the DNS check:
current.cvd.clamav.net<http://current.cvd.clamav.net>. 1800 IN TXT
"0.100.2:58:25048:1539883740:1:63:48006:327"
I.E. 30 minutes for older files, and new ones are when they come in.

Sound about right Joel, Micah?

Sincerely,

Eric Tykwinski
TrueNet, Inc.
P: 610-429-8300

-----Original Message-----
From: clamav-users [mailto:clamav-users-***@lists.clamav.net] On
Behalf Of Paul Kosinski
Sent: Thursday, October 18, 2018 1:23 PM
To: clamav-***@lists.clamav.net<mailto:clamav-***@lists.clamav.net>
Subject: Re: [clamav-users] Latest report on update "delays"

How can it take 10, 20 30 or more minutes (and I've seen well over an
hour at times) to upload the ClamAV database to Cloudflare? Does it have
to be uploaded separately (and maybe sequentially) from Cisco to each
Cloudflare mirror? Or is Cloudflare's automatic propagation slow?


On Thu, 18 Oct 2018 16:07:38 +0000
"Micah Snyder (micasnyd)" <***@cisco.com<mailto:***@cisco.com>> wrote:

Hi Paul,

I realize it may look misleading to state that you're up to date when
a newer database has been announced. However, if the newer database
is still being uploaded to the CDN, it is more accurate to say that
the DNS announcement is premature.

The change to freshclam is an effort to ignore potentially premature
database version numbers listed via DNS.

Micah Snyder
ClamAV Development
Talos
Cisco Systems, Inc.


On Oct 15, 2018, at 2:26 PM, Paul Kosinski
<clamav-***@iment.com<mailto:clamav-***@iment.com><mailto:clamav-***@iment.com>> wrote:

I don't have time at the present to try out 0.100.2. I am rebuilding
our Web server, which had a disk crash. We have backups, but we need
whole new hardware since the old server had an old 32-bit-only CPU.
Thus a *supported* Linux version will not run, and so a simple disk
replacement was not a viable option. (Unfortunately the new server,
although only a VM, still costs almost 50% more per month than the old
raw hardware, which was adequate, if clunky.)

Back to ClamAV: I don't much like the idea of saying signatures are
"up to date" if only 1 version behind the latest version. Most of the
time that won't matter, but sometimes a really urgent new signature
comes out and this approach could mislead people into a false sense of
security.



On Thu, 4 Oct 2018 22:27:14 +0000
"Micah Snyder (micasnyd)"
<***@cisco.com<mailto:***@cisco.com><mailto:***@cisco.com>> wrote:

Hi Paul,

Thanks for the update.

I am interested to know how freshclam in ClamAV 0.100.2 performs for
you. I have made some tweaks to make it ignore mirrors for less
time, but more importantly I implemented a change to have it report
"up to date" in the event that the signature version provided by the
mirror is 1 behind what was advertised. My hope is that this
alleviates the issue.

Respectfully,
Micah


Micah Snyder
ClamAV Development
Talos
Cisco Systems, Inc.


On Oct 4, 2018, at 4:47 PM, Paul Kosinski
<clamav-***@iment.com<mailto:clamav-***@iment.com><mailto:clamav-
***@iment.com<mailto:***@iment.com>><mailto:clamav-***@iment.com>>
wrote:

At Joel's suggestion, i have changed our sampling rate looking for
ClamAV cvd updates from 15 minutes down to 1 minute. This gives a
more precise measurement of how long it takes for the cvd file(s) to
actually become available from Cloudflare after its presence is
"advertised" by the CNS TXT record.

Since these measurements are mainly useful for tuning the ClamAV
servers, I won't in the future post them to clamav-users unless
others besides the ClamAV team find them useful. (Maybe they should
go to the clamav-developers list?)

In any case, here is the latest log of delays. Note that these more
precisely measured delays are not explained as mere 15-minute
quantization errors.

2018-10-02 09:18:02 No delay
2018-10-02 17:18:02 No delay
2018-10-03 01:31:02 00:13:00 delay
2018-10-03 09:42:02 00:24:00 delay
2018-10-03 17:52:02 00:33:59 delay
2018-10-04 01:18:02 No delay
2018-10-04 09:40:01 00:21:59 delay
_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net<mailto:clamav-***@lists.clamav.net><mailto:clamav-
***@lists.clamav.net<mailto:***@lists.clamav.net>><mailto:clamav-***@lists.clamav.net>
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users

_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net<mailto:clamav-***@lists.clamav.net>
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml


_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net<mailto:clamav-***@lists.clamav.net>
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[Eric Tykwinski]

You could limit with Last-Modified, but it’s dependent on the hosting server which CloudFlare can’t control.
Besides, it’s usually just main.cvd that will change mostly and that’s just the first download.

Sincerely,

Eric Tykwinski
TrueNet, Inc.
P: 610-429-8300

> On Oct 19, 2018, at 5:19 PM, Paul Kosinski <clamav-***@iment.com> wrote:
>
> I'm glad modern multi-core / multi-thread CPU's don't operate this way.
>
> Imagine if, when your code on CPU1 tried to access memory location M,
> your code got what CPU1 happened to have in its cache, instead of what
> CPU2 stored into M a few microseconds ago. Fortunately, with real CPUs,
> CPU2 invalidates the other CPUs' caches, and CPU1 takes the extra time
> to fetch the new and correct data from memory.
>
> Thus, what Cloudflare *should* have (if you can't explicitly upload a
> file), is a mechanism to tell it that a file is out of date. This
> mechanism could operate very quickly. Then, what Cloudflare would do is
> either to stall the HTTP response -- I doubt it would have to stall for
> long -- or reply with the appropriate HTTP status code warning the
> requester that something is amiss. (Codes 503, 504 or 409 might be
> applicable.)
>
>
> On Thu, 18 Oct 2018 22:34:03 +0000
> "Joel Esler (jesler)" <***@cisco.com> wrote:
>
>> Cloudflare will grab the file from our infrastructure once it's been
>> requested. (Otherwise it wouldn't know it was there, we can't push
>> into Cloudflare.). But we have discussed a few ideas internally that
>> I think will fix this, let us try a couple things and see if it cuts
>> down on this.
>>
>> On Oct 18, 2018, at 1:55 PM, Eric Tykwinski
>> <eric-***@truenet.com<mailto:eric-***@truenet.com>> wrote:
>>
>> As far as I know you don't upload to cloudflare, it's more of how
>> often does cloudflare check to see if the files have changed.
>> So you setup a TTL on the check frequency on the cloudflare website.
>>
>> Since updates are new they should just be pulled when you ask from
>> the main clam server.
>> So you ask for daily-25048.cdiff, and Cloudflare will ask Clam's main
>> server for that file and cache it.
>>
>> So my guess would be same as the TTL on the DNS check:
>> current.cvd.clamav.net<http://current.cvd.clamav.net>. 1800
>> IN TXT "0.100.2:58:25048:1539883740:1:63:48006:327"
>> I.E. 30 minutes for older files, and new ones are when they come in.
>>
>> Sound about right Joel, Micah?
>>
>> Sincerely,
>>
>> Eric Tykwinski
>> TrueNet, Inc.
>> P: 610-429-8300
>>
>> -----Original Message-----
>> From: clamav-users [mailto:clamav-users-***@lists.clamav.net] On
>> Behalf Of Paul Kosinski
>> Sent: Thursday, October 18, 2018 1:23 PM
>> To:
>> clamav-***@lists.clamav.net<mailto:clamav-***@lists.clamav.net>
>> Subject: Re: [clamav-users] Latest report on update "delays"
>>
>> How can it take 10, 20 30 or more minutes (and I've seen well over an
>> hour at times) to upload the ClamAV database to Cloudflare? Does it
>> have to be uploaded separately (and maybe sequentially) from Cisco to
>> each Cloudflare mirror? Or is Cloudflare's automatic propagation slow?
>>
>>
>> On Thu, 18 Oct 2018 16:07:38 +0000
>> "Micah Snyder (micasnyd)"
>> <***@cisco.com<mailto:***@cisco.com>> wrote:
>>
>> Hi Paul,
>>
>> I realize it may look misleading to state that you're up to date when
>> a newer database has been announced. However, if the newer database
>> is still being uploaded to the CDN, it is more accurate to say that
>> the DNS announcement is premature.
>>
>> The change to freshclam is an effort to ignore potentially premature
>> database version numbers listed via DNS.
>>
>> Micah Snyder
>> ClamAV Development
>> Talos
>> Cisco Systems, Inc.
>>
> _______________________________________________
> clamav-users mailing list
> clamav-***@lists.clamav.net
> http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml

[Joel Esler (jesler)]

They do. There are a bunch of issues with this. Some need to fixed in freshclam, some have to do with cloudflare. The former to be fixed before the latter.

Also, there’s millions of people that need to upgrade to a version that supports cdiffs. And hundreds of thousands that need to use freshclam instead of wget. All of these problems compound.

Sent from my iPhone

> On Oct 19, 2018, at 17:19, Paul Kosinski <clamav-***@iment.com> wrote:
>
> I'm glad modern multi-core / multi-thread CPU's don't operate this way.
>
> Imagine if, when your code on CPU1 tried to access memory location M,
> your code got what CPU1 happened to have in its cache, instead of what
> CPU2 stored into M a few microseconds ago. Fortunately, with real CPUs,
> CPU2 invalidates the other CPUs' caches, and CPU1 takes the extra time
> to fetch the new and correct data from memory.
>
> Thus, what Cloudflare *should* have (if you can't explicitly upload a
> file), is a mechanism to tell it that a file is out of date. This
> mechanism could operate very quickly. Then, what Cloudflare would do is
> either to stall the HTTP response -- I doubt it would have to stall for
> long -- or reply with the appropriate HTTP status code warning the
> requester that something is amiss. (Codes 503, 504 or 409 might be
> applicable.)
>
>
> On Thu, 18 Oct 2018 22:34:03 +0000
> "Joel Esler (jesler)" <***@cisco.com> wrote:
>
>> Cloudflare will grab the file from our infrastructure once it's been
>> requested. (Otherwise it wouldn't know it was there, we can't push
>> into Cloudflare.). But we have discussed a few ideas internally that
>> I think will fix this, let us try a couple things and see if it cuts
>> down on this.
>>
>> On Oct 18, 2018, at 1:55 PM, Eric Tykwinski
>> <eric-***@truenet.com<mailto:eric-***@truenet.com>> wrote:
>>
>> As far as I know you don't upload to cloudflare, it's more of how
>> often does cloudflare check to see if the files have changed.
>> So you setup a TTL on the check frequency on the cloudflare website.
>>
>> Since updates are new they should just be pulled when you ask from
>> the main clam server.
>> So you ask for daily-25048.cdiff, and Cloudflare will ask Clam's main
>> server for that file and cache it.
>>
>> So my guess would be same as the TTL on the DNS check:
>> current.cvd.clamav.net<http://current.cvd.clamav.net>. 1800
>> IN TXT "0.100.2:58:25048:1539883740:1:63:48006:327"
>> I.E. 30 minutes for older files, and new ones are when they come in.
>>
>> Sound about right Joel, Micah?
>>
>> Sincerely,
>>
>> Eric Tykwinski
>> TrueNet, Inc.
>> P: 610-429-8300
>>
>> -----Original Message-----
>> From: clamav-users [mailto:clamav-users-***@lists.clamav.net] On
>> Behalf Of Paul Kosinski
>> Sent: Thursday, October 18, 2018 1:23 PM
>> To:
>> clamav-***@lists.clamav.net<mailto:clamav-***@lists.clamav.net>
>> Subject: Re: [clamav-users] Latest report on update "delays"
>>
>> How can it take 10, 20 30 or more minutes (and I've seen well over an
>> hour at times) to upload the ClamAV database to Cloudflare? Does it
>> have to be uploaded separately (and maybe sequentially) from Cisco to
>> each Cloudflare mirror? Or is Cloudflare's automatic propagation slow?
>>
>>
>> On Thu, 18 Oct 2018 16:07:38 +0000
>> "Micah Snyder (micasnyd)"
>> <***@cisco.com<mailto:***@cisco.com>> wrote:
>>
>> Hi Paul,
>>
>> I realize it may look misleading to state that you're up to date when
>> a newer database has been announced. However, if the newer database
>> is still being uploaded to the CDN, it is more accurate to say that
>> the DNS announcement is premature.
>>
>> The change to freshclam is an effort to ignore potentially premature
>> database version numbers listed via DNS.
>>
>> Micah Snyder
>> ClamAV Development
>> Talos
>> Cisco Systems, Inc.
>>
> _______________________________________________
> clamav-users mailing list
> clamav-***@lists.clamav.net
> http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clam

[Dennis Peterson]

Caching file systems do validate the requested file against a master file to see
if there has been a change. De-dupe caches do the same. It isn't instantaneous
but they also don't have to wait for the cache to refresh as they can deliver a
pass through request at the same time they're updating the cache. This is more
expensive than scheduled sync methods, but those necessarily have a delay. These
systems should reject requests for files they don't have but that is difficult
if the updated file has the same name as the one it replaces. I know it was
always a big deal for the dot com I worked for to update Akamai because of sync
problems around the world. Atomic synchronized file updates are pretty much
impossible when you have a million page requests/minute.

I agree with Joel about using non-standard tools to request signatures and
people that do so should have no expectation of consistent high reliability, and
support requests should go in the bit bucket. The risk associated with
self-service falls on the operator, not the vendor.

dp

On 10/19/18 2:19 PM, Paul Kosinski wrote:
> I'm glad modern multi-core / multi-thread CPU's don't operate this way.
>
> Imagine if, when your code on CPU1 tried to access memory location M,
> your code got what CPU1 happened to have in its cache, instead of what
> CPU2 stored into M a few microseconds ago. Fortunately, with real CPUs,
> CPU2 invalidates the other CPUs' caches, and CPU1 takes the extra time
> to fetch the new and correct data from memory.
>
> Thus, what Cloudflare *should* have (if you can't explicitly upload a
> file), is a mechanism to tell it that a file is out of date. This
> mechanism could operate very quickly. Then, what Cloudflare would do is
> either to stall the HTTP response -- I doubt it would have to stall for
> long -- or reply with the appropriate HTTP status code warning the
> requester that something is amiss. (Codes 503, 504 or 409 might be
> applicable.)
>
>
> On Thu, 18 Oct 2018 22:34:03 +0000
> "Joel Esler (jesler)" <***@cisco.com> wrote:
>
>> Cloudflare will grab the file from our infrastructure once it's been
>> requested. (Otherwise it wouldn't know it was there, we can't push
>> into Cloudflare.). But we have discussed a few ideas internally that
>> I think will fix this, let us try a couple things and see if it cuts
>> down on this.
>>
>> On Oct 18, 2018, at 1:55 PM, Eric Tykwinski
>> <eric-***@truenet.com<mailto:eric-***@truenet.com>> wrote:
>>
>> As far as I know you don't upload to cloudflare, it's more of how
>> often does cloudflare check to see if the files have changed.
>> So you setup a TTL on the check frequency on the cloudflare website.
>>
>> Since updates are new they should just be pulled when you ask from
>> the main clam server.
>> So you ask for daily-25048.cdiff, and Cloudflare will ask Clam's main
>> server for that file and cache it.
>>
>> So my guess would be same as the TTL on the DNS check:
>> current.cvd.clamav.net<http://current.cvd.clamav.net>. 1800
>> IN TXT "0.100.2:58:25048:1539883740:1:63:48006:327"
>> I.E. 30 minutes for older files, and new ones are when they come in.
>>
>> Sound about right Joel, Micah?
>>
>> Sincerely,
>>
>> Eric Tykwinski
>> TrueNet, Inc.
>> P: 610-429-8300
>>
>> -----Original Message-----
>> From: clamav-users [mailto:clamav-users-***@lists.clamav.net] On
>> Behalf Of Paul Kosinski
>> Sent: Thursday, October 18, 2018 1:23 PM
>> To:
>> clamav-***@lists.clamav.net<mailto:clamav-***@lists.clamav.net>
>> Subject: Re: [clamav-users] Latest report on update "delays"
>>
>> How can it take 10, 20 30 or more minutes (and I've seen well over an
>> hour at times) to upload the ClamAV database to Cloudflare? Does it
>> have to be uploaded separately (and maybe sequentially) from Cisco to
>> each Cloudflare mirror? Or is Cloudflare's automatic propagation slow?
>>
>>
>> On Thu, 18 Oct 2018 16:07:38 +0000
>> "Micah Snyder (micasnyd)"
>> <***@cisco.com<mailto:***@cisco.com>> wrote:
>>
>> Hi Paul,
>>
>> I realize it may look misleading to state that you're up to date when
>> a newer database has been announced. However, if the newer database
>> is still being uploaded to the CDN, it is more accurate to say that
>> the DNS announcement is premature.
>>
>> The change to freshclam is an effort to ignore potentially premature
>> database version numbers listed via DNS.
>>
>> Micah Snyder
>> ClamAV Development
>> Talos
>> Cisco Systems, Inc.
>>
> _______________________________________________
> clamav-users mailing list
> clamav-***@lists.clamav.net
> http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml


_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[Joel Esler (jesler)]

Right Dennis. There will always be a delay, I can’t get it down to zero. But I can minimize it. That being said, the infrastructure we are using today is a ton more reliable than the previous mirror infrastructure.

Sent from my iPhone

> On Oct 20, 2018, at 09:58, Dennis Peterson <***@inetnw.com> wrote:
>
> Caching file systems do validate the requested file against a master file to see if there has been a change. De-dupe caches do the same. It isn't instantaneous but they also don't have to wait for the cache to refresh as they can deliver a pass through request at the same time they're updating the cache. This is more expensive than scheduled sync methods, but those necessarily have a delay. These systems should reject requests for files they don't have but that is difficult if the updated file has the same name as the one it replaces. I know it was always a big deal for the dot com I worked for to update Akamai because of sync problems around the world. Atomic synchronized file updates are pretty much impossible when you have a million page requests/minute.
>
> I agree with Joel about using non-standard tools to request signatures and people that do so should have no expectation of consistent high reliability, and support requests should go in the bit bucket. The risk associated with self-service falls on the operator, not the vendor.
>
> dp
>
>> On 10/19/18 2:19 PM, Paul Kosinski wrote:
>> I'm glad modern multi-core / multi-thread CPU's don't operate this way.
>>
>> Imagine if, when your code on CPU1 tried to access memory location M,
>> your code got what CPU1 happened to have in its cache, instead of what
>> CPU2 stored into M a few microseconds ago. Fortunately, with real CPUs,
>> CPU2 invalidates the other CPUs' caches, and CPU1 takes the extra time
>> to fetch the new and correct data from memory.
>>
>> Thus, what Cloudflare *should* have (if you can't explicitly upload a
>> file), is a mechanism to tell it that a file is out of date. This
>> mechanism could operate very quickly. Then, what Cloudflare would do is
>> either to stall the HTTP response -- I doubt it would have to stall for
>> long -- or reply with the appropriate HTTP status code warning the
>> requester that something is amiss. (Codes 503, 504 or 409 might be
>> applicable.)
>>
>>
>> On Thu, 18 Oct 2018 22:34:03 +0000
>> "Joel Esler (jesler)" <***@cisco.com> wrote:
>>
>>> Cloudflare will grab the file from our infrastructure once it's been
>>> requested. (Otherwise it wouldn't know it was there, we can't push
>>> into Cloudflare.). But we have discussed a few ideas internally that
>>> I think will fix this, let us try a couple things and see if it cuts
>>> down on this.
>>>
>>> On Oct 18, 2018, at 1:55 PM, Eric Tykwinski
>>> <eric-***@truenet.com<mailto:eric-***@truenet.com>> wrote:
>>>
>>> As far as I know you don't upload to cloudflare, it's more of how
>>> often does cloudflare check to see if the files have changed.
>>> So you setup a TTL on the check frequency on the cloudflare website.
>>>
>>> Since updates are new they should just be pulled when you ask from
>>> the main clam server.
>>> So you ask for daily-25048.cdiff, and Cloudflare will ask Clam's main
>>> server for that file and cache it.
>>>
>>> So my guess would be same as the TTL on the DNS check:
>>> current.cvd.clamav.net<http://current.cvd.clamav.net>. 1800
>>> IN TXT "0.100.2:58:25048:1539883740:1:63:48006:327"
>>> I.E. 30 minutes for older files, and new ones are when they come in.
>>>
>>> Sound about right Joel, Micah?
>>>
>>> Sincerely,
>>>
>>> Eric Tykwinski
>>> TrueNet, Inc.
>>> P: 610-429-8300
>>>
>>> -----Original Message-----
>>> From: clamav-users [mailto:clamav-users-***@lists.clamav.net] On
>>> Behalf Of Paul Kosinski
>>> Sent: Thursday, October 18, 2018 1:23 PM
>>> To:
>>> clamav-***@lists.clamav.net<mailto:clamav-***@lists.clamav.net>
>>> Subject: Re: [clamav-users] Latest report on update "delays"
>>>
>>> How can it take 10, 20 30 or more minutes (and I've seen well over an
>>> hour at times) to upload the ClamAV database to Cloudflare? Does it
>>> have to be uploaded separately (and maybe sequentially) from Cisco to
>>> each Cloudflare mirror? Or is Cloudflare's automatic propagation slow?
>>>
>>>
>>> On Thu, 18 Oct 2018 16:07:38 +0000
>>> "Micah Snyder (micasnyd)"
>>> <***@cisco.com<mailto:***@cisco.com>> wrote:
>>>
>>> Hi Paul,
>>>
>>> I realize it may look misleading to state that you're up to date when
>>> a newer database has been announced. However, if the newer database
>>> is still being uploaded to the CDN, it is more accurate to say that
>>> the DNS announcement is premature.
>>>
>>> The change to freshclam is an effort to ignore potentially premature
>>> database version numbers listed via DNS.
>>>
>>> Micah Snyder
>>> ClamAV Development
>>> Talos
>>> Cisco Systems, Inc.
>>>
>> _______________________________________________
>> clamav-users mailing list
>> clamav-***@lists.clamav.net
>> http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
>>
>>
>> Help us build a comprehensive ClamAV guide:
>> https://github.com/vrtadmin/clamav-faq
>>
>> http://www.clamav.net/contact.html#ml
>
>
> _______________________________________________
> clamav-users mailing list
> clamav-***@lists.clamav.net
> http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav

[Joel Esler (jesler)]

Well the TXT based approach is to tell you (the users) when to update. It’s a decent approach, and any change to that would be a good amount of work. Which we’re not against, it’s just a baby step. It also means, (which may be a good thing?) we’d have a departure between “people who keep ClamAV up to date, and people that are attempting to update version 0.88”.

Interesting debate topics for our meetings.

I think, any transition we do, has to be slow. We went through a “clean break” transition with the Snort project many years ago when we shifted from one way of doing things to another, and we got a lot of pushback when we did it. So this is one of the few times when I don’t want to rip off the bandaid and just make a change.

This will involve a lot of coordination, not only with open source, but with internal (to Cisco) people that use ClamAV in various products, the development team lead by Cisco Talos (who coordinates very well with those internal stakeholders), which people like Micah are heavily involved. But also with external (to Cisco) stakeholders like everyone on this list and beyond, as ClamAV is built into countless products.

Please keep providing feedback. That helps me and others scope the forward direction.

Sent from my iPhone

> On Oct 20, 2018, at 11:10, Paul Kosinski <clamav-***@iment.com> wrote:
>
> Yes, file synchronization is difficult. But we *started out* using the
> provided (i.e., standard) freshclam tool to update our daily.cvd (etc.).
> I only built our current non-standard tool (reading the file header)
> when the Cloudflare mirrors started serving out-of-date file versions
> which caused freshclam to fail and blacklist the mirror (which
> eventually resulted in all mirrors being blacklisted).
>
> This says to me that the old, "standard", DNS TXT approach built in to
> freshclam doesn't play well with Cloudflare (or similar mirrors?).
>
>
>
> On Sat, 20 Oct 2018 06:57:55 -0700
> Dennis Peterson <***@inetnw.com> wrote:
>
>> Caching file systems do validate the requested file against a master
>> file to see if there has been a change. De-dupe caches do the same.
>> It isn't instantaneous but they also don't have to wait for the cache
>> to refresh as they can deliver a pass through request at the same
>> time they're updating the cache. This is more expensive than
>> scheduled sync methods, but those necessarily have a delay. These
>> systems should reject requests for files they don't have but that is
>> difficult if the updated file has the same name as the one it
>> replaces. I know it was always a big deal for the dot com I worked
>> for to update Akamai because of sync problems around the world.
>> Atomic synchronized file updates are pretty much impossible when you
>> have a million page requests/minute.
>>
>> I agree with Joel about using non-standard tools to request
>> signatures and people that do so should have no expectation of
>> consistent high reliability, and support requests should go in the
>> bit bucket. The risk associated with self-service falls on the
>> operator, not the vendor.
>>
>> dp
>>
>>> On 10/19/18 2:19 PM, Paul Kosinski wrote:
>>> I'm glad modern multi-core / multi-thread CPU's don't operate this
>>> way.
>>>
>>> Imagine if, when your code on CPU1 tried to access memory location
>>> M, your code got what CPU1 happened to have in its cache, instead
>>> of what CPU2 stored into M a few microseconds ago. Fortunately,
>>> with real CPUs, CPU2 invalidates the other CPUs' caches, and CPU1
>>> takes the extra time to fetch the new and correct data from memory.
>>>
>>> Thus, what Cloudflare *should* have (if you can't explicitly upload
>>> a file), is a mechanism to tell it that a file is out of date. This
>>> mechanism could operate very quickly. Then, what Cloudflare would
>>> do is either to stall the HTTP response -- I doubt it would have to
>>> stall for long -- or reply with the appropriate HTTP status code
>>> warning the requester that something is amiss. (Codes 503, 504 or
>>> 409 might be applicable.)
>>>
>>>
>>> On Thu, 18 Oct 2018 22:34:03 +0000
>>> "Joel Esler (jesler)" <***@cisco.com> wrote:
>>>
>>>> Cloudflare will grab the file from our infrastructure once it's
>>>> been requested. (Otherwise it wouldn't know it was there, we
>>>> can't push into Cloudflare.). But we have discussed a few ideas
>>>> internally that I think will fix this, let us try a couple things
>>>> and see if it cuts down on this.
>>>>
>>>> On Oct 18, 2018, at 1:55 PM, Eric Tykwinski
>>>> <eric-***@truenet.com<mailto:eric-***@truenet.com>> wrote:
>>>>
>>>> As far as I know you don't upload to cloudflare, it's more of how
>>>> often does cloudflare check to see if the files have changed.
>>>> So you setup a TTL on the check frequency on the cloudflare
>>>> website.
>>>>
>>>> Since updates are new they should just be pulled when you ask from
>>>> the main clam server.
>>>> So you ask for daily-25048.cdiff, and Cloudflare will ask Clam's
>>>> main server for that file and cache it.
>>>>
>>>> So my guess would be same as the TTL on the DNS check:
>>>> current.cvd.clamav.net<http://current.cvd.clamav.net>. 1800
>>>> IN TXT "0.100.2:58:25048:1539883740:1:63:48006:327"
>>>> I.E. 30 minutes for older files, and new ones are when they come
>>>> in.
>>>>
>>>> Sound about right Joel, Micah?
>>>>
>>>> Sincerely,
>>>>
>>>> Eric Tykwinski
>>>> TrueNet, Inc.
>>>> P: 610-429-8300
>>>>
>>>> -----Original Message-----
>>>> From: clamav-users [mailto:clamav-users-***@lists.clamav.net]
>>>> On Behalf Of Paul Kosinski
>>>> Sent: Thursday, October 18, 2018 1:23 PM
>>>> To:
>>>> clamav-***@lists.clamav.net<mailto:clamav-***@lists.clamav.net>
>>>> Subject: Re: [clamav-users] Latest report on update "delays"
>>>>
>>>> How can it take 10, 20 30 or more minutes (and I've seen well over
>>>> an hour at times) to upload the ClamAV database to Cloudflare?
>>>> Does it have to be uploaded separately (and maybe sequentially)
>>>> from Cisco to each Cloudflare mirror? Or is Cloudflare's automatic
>>>> propagation slow?
>>>>
>>>>
>>>> On Thu, 18 Oct 2018 16:07:38 +0000
>>>> "Micah Snyder (micasnyd)"
>>>> <***@cisco.com<mailto:***@cisco.com>> wrote:
>>>>
>>>> Hi Paul,
>>>>
>>>> I realize it may look misleading to state that you're up to date
>>>> when a newer database has been announced. However, if the newer
>>>> database is still being uploaded to the CDN, it is more accurate
>>>> to say that the DNS announcement is premature.
>>>>
>>>> The change to freshclam is an effort to ignore potentially
>>>> premature database version numbers listed via DNS.
>>>>
>>>> Micah Snyder
>>>> ClamAV Development
>>>> Talos
>>>> Cisco Systems, Inc.
>
> _______________________________________________
> clamav-users mailing list
> clamav-***@lists.clamav.net
> http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/cla

[Dennis Peterson]

You should abandon the notion of first time perfect with these kinds of things.
There is a false sense of urgency that is imposing a workload on a team that is
providing a free produce and service. The tools for correcting a moment zero
malware exists in the tool for the operator to use. The real problem is the
discovery and validation and that is why moment zero solutions will never be
possible.

There is a finite time required to receive a malware instance, discover it is a
malware, discover what it applies to, and to create a signature that reasonably
avoids false positives. I'm convinced that interval exceeds the delay due to
sync problems by such a margin that the first interval needs as much focus as
can be committed while the distribution issues are handled at a lower priority.

There are other probabilities - as an example, the probability that a new
malware is sufficiently in the wild to pose a threat to an important number of
recipients and which can be very low. Those can be queued for release cutting
down on the number of low-value updates. And somebody has to decide what is an
important number.

Evidence of self-replication is recognizable by the rate of increase of
infestations and is data that can be used in setting priorities. How to collect
that? How to collect any metrics? So far it is largely buzz generated by
responders and which is largely anecdotal.

To be honest, many problems would be solved if all outbound mail were scanned in
real time.

dp

On 10/20/18 8:10 AM, Paul Kosinski wrote:
> Yes, file synchronization is difficult. But we *started out* using the
> provided (i.e., standard) freshclam tool to update our daily.cvd (etc.).
> I only built our current non-standard tool (reading the file header)
> when the Cloudflare mirrors started serving out-of-date file versions
> which caused freshclam to fail and blacklist the mirror (which
> eventually resulted in all mirrors being blacklisted).
>
> This says to me that the old, "standard", DNS TXT approach built in to
> freshclam doesn't play well with Cloudflare (or similar mirrors?).
>
>
>
> On Sat, 20 Oct 2018 06:57:55 -0700
> Dennis Peterson <***@inetnw.com> wrote:
>
>> Caching file systems do validate the requested file against a master
>> file to see if there has been a change. De-dupe caches do the same.
>> It isn't instantaneous but they also don't have to wait for the cache
>> to refresh as they can deliver a pass through request at the same
>> time they're updating the cache. This is more expensive than
>> scheduled sync methods, but those necessarily have a delay. These
>> systems should reject requests for files they don't have but that is
>> difficult if the updated file has the same name as the one it
>> replaces. I know it was always a big deal for the dot com I worked
>> for to update Akamai because of sync problems around the world.
>> Atomic synchronized file updates are pretty much impossible when you
>> have a million page requests/minute.
>>
>> I agree with Joel about using non-standard tools to request
>> signatures and people that do so should have no expectation of
>> consistent high reliability, and support requests should go in the
>> bit bucket. The risk associated with self-service falls on the
>> operator, not the vendor.
>>
>> dp
>>
>> On 10/19/18 2:19 PM, Paul Kosinski wrote:
>>> I'm glad modern multi-core / multi-thread CPU's don't operate this
>>> way.
>>>
>>> Imagine if, when your code on CPU1 tried to access memory location
>>> M, your code got what CPU1 happened to have in its cache, instead
>>> of what CPU2 stored into M a few microseconds ago. Fortunately,
>>> with real CPUs, CPU2 invalidates the other CPUs' caches, and CPU1
>>> takes the extra time to fetch the new and correct data from memory.
>>>
>>> Thus, what Cloudflare *should* have (if you can't explicitly upload
>>> a file), is a mechanism to tell it that a file is out of date. This
>>> mechanism could operate very quickly. Then, what Cloudflare would
>>> do is either to stall the HTTP response -- I doubt it would have to
>>> stall for long -- or reply with the appropriate HTTP status code
>>> warning the requester that something is amiss. (Codes 503, 504 or
>>> 409 might be applicable.)
>>>
>>>
>>> On Thu, 18 Oct 2018 22:34:03 +0000
>>> "Joel Esler (jesler)" <***@cisco.com> wrote:
>>>
>>>> Cloudflare will grab the file from our infrastructure once it's
>>>> been requested. (Otherwise it wouldn't know it was there, we
>>>> can't push into Cloudflare.). But we have discussed a few ideas
>>>> internally that I think will fix this, let us try a couple things
>>>> and see if it cuts down on this.
>>>>
>>>> On Oct 18, 2018, at 1:55 PM, Eric Tykwinski
>>>> <eric-***@truenet.com<mailto:eric-***@truenet.com>> wrote:
>>>>
>>>> As far as I know you don't upload to cloudflare, it's more of how
>>>> often does cloudflare check to see if the files have changed.
>>>> So you setup a TTL on the check frequency on the cloudflare
>>>> website.
>>>>
>>>> Since updates are new they should just be pulled when you ask from
>>>> the main clam server.
>>>> So you ask for daily-25048.cdiff, and Cloudflare will ask Clam's
>>>> main server for that file and cache it.
>>>>
>>>> So my guess would be same as the TTL on the DNS check:
>>>> current.cvd.clamav.net<http://current.cvd.clamav.net>. 1800
>>>> IN TXT "0.100.2:58:25048:1539883740:1:63:48006:327"
>>>> I.E. 30 minutes for older files, and new ones are when they come
>>>> in.
>>>>
>>>> Sound about right Joel, Micah?
>>>>
>>>> Sincerely,
>>>>
>>>> Eric Tykwinski
>>>> TrueNet, Inc.
>>>> P: 610-429-8300
>>>>
>>>> -----Original Message-----
>>>> From: clamav-users [mailto:clamav-users-***@lists.clamav.net]
>>>> On Behalf Of Paul Kosinski
>>>> Sent: Thursday, October 18, 2018 1:23 PM
>>>> To:
>>>> clamav-***@lists.clamav.net<mailto:clamav-***@lists.clamav.net>
>>>> Subject: Re: [clamav-users] Latest report on update "delays"
>>>>
>>>> How can it take 10, 20 30 or more minutes (and I've seen well over
>>>> an hour at times) to upload the ClamAV database to Cloudflare?
>>>> Does it have to be uploaded separately (and maybe sequentially)
>>>> from Cisco to each Cloudflare mirror? Or is Cloudflare's automatic
>>>> propagation slow?
>>>>
>>>>
>>>> On Thu, 18 Oct 2018 16:07:38 +0000
>>>> "Micah Snyder (micasnyd)"
>>>> <***@cisco.com<mailto:***@cisco.com>> wrote:
>>>>
>>>> Hi Paul,
>>>>
>>>> I realize it may look misleading to state that you're up to date
>>>> when a newer database has been announced. However, if the newer
>>>> database is still being uploaded to the CDN, it is more accurate
>>>> to say that the DNS announcement is premature.
>>>>
>>>> The change to freshclam is an effort to ignore potentially
>>>> premature database version numbers listed via DNS.
>>>>
>>>> Micah Snyder
>>>> ClamAV Development
>>>> Talos
>>>> Cisco Systems, Inc.
> _______________________________________________
> clamav-users mailing list
> clamav-***@lists.clamav.net
> http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml


_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[Gary R. Schmidt]

On 23/10/2018 13:28, Paul Kosinski wrote:
> "I'm convinced that [malware analysis] interval exceeds the delay due to
> sync problems by such a margin that the first interval needs as much
> focus as can be committed while the distribution issues are handled at
> a lower priority."
>
> I mainly agree (and I much appreciate the efforts of the ClamAV team).
>
> What we found, unfortunately, was that after the switch to Cloudflare,
> the mirror sync problems observed by "stock" freshclam resulted in all
> the mirrors being blacklisted, causing future ClamAV virus updates to
> cease. This meant distribution issues became extremely important.
>
Would just adding a cron job that deletes "mirrors.dat" every so often
be an acceptable work-around?

It amuses me that my mirrors.dat file contains five entries that all
point to the same IP address, that's just a bit pointless!

Cheers,
Gary B-)
_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[Gary R. Schmidt]

On 23/10/2018 16:17, Paul Kosinski wrote:
> Two observations: First, a smoothly working freshclam mechanism
> shouldn't require workarounds.
Well, yes, but it works smoothly for a very large number of people,
myself included.

> And I suspect many ClamAV users
> wouldn't be able to deal with workarounds like this.
This I disagree with - if they are bright enough to go looking for
something like ClamAV then they'd be able to handle the trivial task of
adding a line to a crontab file.

> Second, any time freshclam fails due to an out-of-sync problem, there
> has been a useless load on the mirrors (although I suppose using cdiffs
> would significantly reduce the useless data transfer). Plus there is
> useless load on the client machine and its LAN.
There is a load only on the mirrors you are accessing, the rest of them
keep doing their job.

Have you absolutely ruled out the possibility of someone having set up a
transparent proxy on your border router(s)?

Cheers,
Gary B-)
_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[Joel Esler (jesler)]

We are aware that fresh clam is part of the issue. We are going to introduce some new code to freshclam (and have in the past two releases, IIRC) to prevent stuff like this happening. More updates to freshclam will come in future versions as well.

That being said, it's important to realize that we will still have millions of users stuck on old version. But there is only so much we can do for those people.

> On Oct 23, 2018, at 1:52 AM, Gary R. Schmidt <***@acm.org> wrote:
>
> On 23/10/2018 16:17, Paul Kosinski wrote:
>> Two observations: First, a smoothly working freshclam mechanism
>> shouldn't require workarounds.
> Well, yes, but it works smoothly for a very large number of people, myself included.
>
>> And I suspect many ClamAV users
>> wouldn't be able to deal with workarounds like this.
> This I disagree with - if they are bright enough to go looking for something like ClamAV then they'd be able to handle the trivial task of adding a line to a crontab file.
>
>> Second, any time freshclam fails due to an out-of-sync problem, there
>> has been a useless load on the mirrors (although I suppose using cdiffs
>> would significantly reduce the useless data transfer). Plus there is
>> useless load on the client machine and its LAN.
> There is a load only on the mirrors you are accessing, the rest of them keep doing their job.
>
> Have you absolutely ruled out the possibility of someone having set up a transparent proxy on your border router(s)?
>
> Cheers,
> Gary B-)
> _______________________________________________
> clamav-users mailing list
> clamav-***@lists.clamav.net
> http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml

_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[Dave Warren]

On Tue, Oct 23, 2018, at 11:50, Paul Kosinski wrote:
> "...it works smoothly for a very large number of people, myself
> included."
>
> It would be interesting to know what percentage have experienced our
> original problem of all mirrors ending up blacklisted. I also wonder
> how many ClamAV users monitor their logs: I don't remember ClamAV
> *actively* reporting when signatures are out date (like some
> Windows AV does).
>
>
> "Have you absolutely ruled out the possibility of someone having set up
> a transparent proxy on your border router(s)?"
>
> I guess I wouldn't put it past Comcast/Xfinity to interpose a proxy,
> but if the proxy is transparent, how would one detect it? And would a
> proxy once in a while be over an hour out of date? Maybe nobody in
> *this* part of the Boston area uses ClamAV, but Boston is not exactly
> Ulaanbaatar Mongolia (an actual Cloudflare mirror!).

http://www.lagado.com/tools/cache-test can be used to detect caching (transparent or otherwise).

_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[Michael Da Cova]

Hi

On 24/10/2018 04:09, Dave Warren wrote:
> On Tue, Oct 23, 2018, at 11:50, Paul Kosinski wrote:
>> "...it works smoothly for a very large number of people, myself
>> included."
>>
>> It would be interesting to know what percentage have experienced our
>> original problem of all mirrors ending up blacklisted.

I still get the issue now and again, today report below if I notice it I
remove the mirror.dat file

Retrieving http://database.clamav.net/daily.cvd
Ignoring mirror 104.16.187.138 (due to previous errors)
Ignoring mirror 104.16.188.138 (due to previous errors)
Ignoring mirror 104.16.185.138 (due to previous errors)
Ignoring mirror 104.16.186.138 (due to previous errors)
Ignoring mirror 104.16.189.138 (due to previous errors)
Trying host database.clamav.net (2400:cb00:2048:1::6810:ba8a)...
nonblock_connect: connect(): fd=4 errno=101: Network is unreachable

~michael
_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[Joel Esler (jesler)]

If you are testing connectivity, please state what version of ClamAV you are using.

If you are not using the most up to date, please try that.

Sent from my iPhone

> On Oct 24, 2018, at 04:00, Michael Da Cova <***@netpilot.com> wrote:
>
> Hi
>
>> On 24/10/2018 04:09, Dave Warren wrote:
>>> On Tue, Oct 23, 2018, at 11:50, Paul Kosinski wrote:
>>> "...it works smoothly for a very large number of people, myself
>>> included."
>>>
>>> It would be interesting to know what percentage have experienced our
>>> original problem of all mirrors ending up blacklisted.
>
> I still get the issue now and again, today report below if I notice it I
> remove the mirror.dat file
>
> Retrieving http://database.clamav.net/daily.cvd
> Ignoring mirror 104.16.187.138 (due to previous errors)
> Ignoring mirror 104.16.188.138 (due to previous errors)
> Ignoring mirror 104.16.185.138 (due to previous errors)
> Ignoring mirror 104.16.186.138 (due to previous errors)
> Ignoring mirror 104.16.189.138 (due to previous errors)
> Trying host database.clamav.net (2400:cb00:2048:1::6810:ba8a)...
> nonblock_connect: connect(): fd=4 errno=101: Network is unreachable
>
> ~michael
> _______________________________________________
> clamav-users mailing list
> clamav-***@lists.clamav.net
> http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml