[clamav-users] Still over 1/3 signature update sync errors

[Joel Esler (jesler)]

Thanks Paul, this is super useful.

> On Jul 31, 2018, at 1:47 PM, Paul Kosinski <clamav-***@iment.com> wrote:
>
> There are still over 1/3 signature update sync errors with the new
> ClamAV mirrors.
>
> You may remember that I previously added code to our ClamAV update
> protocol to verify that the actually available daily.cvd etc. matched
> the version number reported by the DNS TXT record. (This is done by
> using curl to retrieve a short prefix of the daily.cvd etc. files.)
>
> Some days ago I also added code to summarize any synchronization
> problems. This code logs how much, if any, delay obtains between the
> DNS TXT record reporting a new version and curl agreeing that the new
> version is actually available from a mirror. This protocol operates
> every 15 minutes (giving rise to a rounding error of up to 15 minutes).
>
> Recent results are as follows:
>
> 2018-07-26 05:18:02 No delay
> 2018-07-26 13:18:02 No delay
> 2018-07-26 13:33:01 No delay
> 2018-07-26 13:48:01 No delay
> 2018-07-26 14:03:01 No delay
> 2018-07-26 14:18:02 No delay
> 2018-07-26 21:33:02 00:45:00 delay
> 2018-07-27 05:03:02 No delay
> 2018-07-27 13:18:02 No delay
> 2018-07-27 13:18:02 No delay
> 2018-07-27 21:34:05 No delay
> 2018-07-27 21:34:05 No delay
> 2018-07-28 05:48:02 00:30:01 delay
> 2018-07-28 13:18:02 No delay
> 2018-07-28 21:18:02 00:30:01 delay
> 2018-07-29 05:33:01 00:29:59 delay
> 2018-07-29 14:18:02 01:30:00 delay
> 2018-07-29 21:18:02 00:30:00 delay
> 2018-07-30 04:48:02 No delay
> 2018-07-30 13:33:01 00:44:59 delay
> 2018-07-30 21:48:02 01:00:00 delay
> 2018-07-31 05:18:02 No delay
> 2018-07-31 13:33:02 No delay
>
> Note: The ClamAV update protocol uses the URL "database.clamav.net",
> which in turn resolves to 1 of 5 IP addresses, which in turn route to
> one of many geographically separated systems. Thus, the delays may not
> correspond to problems with particular paths and/or mirror instances.
>
>
> P.S. I can provide a more detailed freshclam log for this if desired.
> _______________________________________________
> clamav-users mailing list
> clamav-***@lists.clamav.net
> http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml

_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[Joel Esler (jesler)]

I actually just made an adjustment today to see if that will resolve the issues. Please keep these coming?!

Sent from my iPad

> On Aug 11, 2018, at 2:10 PM, Paul Kosinski <clamav-***@iment.com> wrote:
>
> Here is the latest report for ClamAV virus update mirror delays since
> the end of July. DNS TXT vs actual file availability has gotten worse!
> Now, over 57% of updates are DNS-announced prematurely. (Last time, only
> about 1/3 were announced prematurely.)
>
> 2018-07-31 21:33:01 No delay
> 2018-08-01 05:03:01 00:14:59 delay
> 2018-08-01 13:18:01 No delay
> 2018-08-01 21:03:02 00:15:00 delay
> 2018-08-02 05:18:02 No delay
> 2018-08-02 13:48:02 00:30:00 delay
> 2018-08-02 21:48:01 00:15:00 delay
> 2018-08-03 05:18:02 No delay
> 2018-08-03 14:18:02 00:30:01 delay
> 2018-08-03 21:48:02 00:30:01 delay
> 2018-08-04 05:18:01 No delay
> 2018-08-04 13:18:02 No delay
> 2018-08-04 21:33:01 00:14:59 delay
> 2018-08-05 05:48:02 01:00:00 delay
> 2018-08-05 13:03:02 00:15:00 delay
> 2018-08-05 21:03:02 00:15:00 delay
> 2018-08-06 05:18:02 No delay
> 2018-08-06 13:18:02 No delay
> 2018-08-07 00:03:01 02:45:00 delay
> 2018-08-07 05:18:02 No delay
> 2018-08-07 13:18:02 No delay
> 2018-08-07 21:48:01 00:29:59 delay
> 2018-08-08 05:48:01 01:00:00 delay
> 2018-08-08 13:48:02 00:30:01 delay
> 2018-08-08 21:18:01 No delay
> 2018-08-09 05:33:02 No delay
> 2018-08-09 14:33:01 01:15:00 delay
> 2018-08-09 22:33:01 00:44:59 delay
> 2018-08-10 05:48:01 01:00:00 delay
> 2018-08-10 13:48:02 00:30:00 delay
> 2018-08-11 00:48:02 03:30:00 delay
> 2018-08-11 05:33:02 No delay
> 2018-08-11 13:18:02 No delay
>
>
>
>
> On Tue, 31 Jul 2018 13:47:39 -0400
> Paul Kosinski <clamav-***@iment.com> wrote:
>
>> There are still over 1/3 signature update sync errors with the new
>> ClamAV mirrors.
>>
>> You may remember that I previously added code to our ClamAV update
>> protocol to verify that the actually available daily.cvd etc. matched
>> the version number reported by the DNS TXT record. (This is done by
>> using curl to retrieve a short prefix of the daily.cvd etc. files.)
>>
>> Some days ago I also added code to summarize any synchronization
>> problems. This code logs how much, if any, delay obtains between the
>> DNS TXT record reporting a new version and curl agreeing that the new
>> version is actually available from a mirror. This protocol operates
>> every 15 minutes (giving rise to a rounding error of up to 15
>> minutes).
>>
>> Recent results are as follows:
>>
>> 2018-07-26 05:18:02 No delay
>> 2018-07-26 13:18:02 No delay
>> 2018-07-26 13:33:01 No delay
>> 2018-07-26 13:48:01 No delay
>> 2018-07-26 14:03:01 No delay
>> 2018-07-26 14:18:02 No delay
>> 2018-07-26 21:33:02 00:45:00 delay
>> 2018-07-27 05:03:02 No delay
>> 2018-07-27 13:18:02 No delay
>> 2018-07-27 13:18:02 No delay
>> 2018-07-27 21:34:05 No delay
>> 2018-07-27 21:34:05 No delay
>> 2018-07-28 05:48:02 00:30:01 delay
>> 2018-07-28 13:18:02 No delay
>> 2018-07-28 21:18:02 00:30:01 delay
>> 2018-07-29 05:33:01 00:29:59 delay
>> 2018-07-29 14:18:02 01:30:00 delay
>> 2018-07-29 21:18:02 00:30:00 delay
>> 2018-07-30 04:48:02 No delay
>> 2018-07-30 13:33:01 00:44:59 delay
>> 2018-07-30 21:48:02 01:00:00 delay
>> 2018-07-31 05:18:02 No delay
>> 2018-07-31 13:33:02 No delay
>>
>> Note: The ClamAV update protocol uses the URL "database.clamav.net",
>> which in turn resolves to 1 of 5 IP addresses, which in turn route to
>> one of many geographically separated systems. Thus, the delays may not
>> correspond to problems with particular paths and/or mirror instances.
>>
>>
>> P.S. I can provide a more detailed freshclam log for this if desired.
>
>>
> _______________________________________________
> clamav-users mailing list
> clamav-***@lists.clamav.net
> http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[Joel Esler (jesler)]

Paul, how are things looking from your side?

--
Joel Esler
Sr. Manager
Community, Branding, and Open Source
Talos Group
http://www.talosintelligence.com

On Aug 11, 2018, at 6:12 PM, Joel Esler (jesler) <***@cisco.com<mailto:***@cisco.com>> wrote:

I actually just made an adjustment today to see if that will resolve the issues. Please keep these coming?!

Sent from my iPad

On Aug 11, 2018, at 2:10 PM, Paul Kosinski <clamav-***@iment.com<mailto:clamav-***@iment.com>> wrote:

Here is the latest report for ClamAV virus update mirror delays since
the end of July. DNS TXT vs actual file availability has gotten worse!
Now, over 57% of updates are DNS-announced prematurely. (Last time, only
about 1/3 were announced prematurely.)

2018-07-31 21:33:01 No delay
2018-08-01 05:03:01 00:14:59 delay
2018-08-01 13:18:01 No delay
2018-08-01 21:03:02 00:15:00 delay
2018-08-02 05:18:02 No delay
2018-08-02 13:48:02 00:30:00 delay
2018-08-02 21:48:01 00:15:00 delay
2018-08-03 05:18:02 No delay
2018-08-03 14:18:02 00:30:01 delay
2018-08-03 21:48:02 00:30:01 delay
2018-08-04 05:18:01 No delay
2018-08-04 13:18:02 No delay
2018-08-04 21:33:01 00:14:59 delay
2018-08-05 05:48:02 01:00:00 delay
2018-08-05 13:03:02 00:15:00 delay
2018-08-05 21:03:02 00:15:00 delay
2018-08-06 05:18:02 No delay
2018-08-06 13:18:02 No delay
2018-08-07 00:03:01 02:45:00 delay
2018-08-07 05:18:02 No delay
2018-08-07 13:18:02 No delay
2018-08-07 21:48:01 00:29:59 delay
2018-08-08 05:48:01 01:00:00 delay
2018-08-08 13:48:02 00:30:01 delay
2018-08-08 21:18:01 No delay
2018-08-09 05:33:02 No delay
2018-08-09 14:33:01 01:15:00 delay
2018-08-09 22:33:01 00:44:59 delay
2018-08-10 05:48:01 01:00:00 delay
2018-08-10 13:48:02 00:30:00 delay
2018-08-11 00:48:02 03:30:00 delay
2018-08-11 05:33:02 No delay
2018-08-11 13:18:02 No delay




On Tue, 31 Jul 2018 13:47:39 -0400
Paul Kosinski <clamav-***@iment.com<mailto:clamav-***@iment.com>> wrote:

There are still over 1/3 signature update sync errors with the new
ClamAV mirrors.

You may remember that I previously added code to our ClamAV update
protocol to verify that the actually available daily.cvd etc. matched
the version number reported by the DNS TXT record. (This is done by
using curl to retrieve a short prefix of the daily.cvd etc. files.)

Some days ago I also added code to summarize any synchronization
problems. This code logs how much, if any, delay obtains between the
DNS TXT record reporting a new version and curl agreeing that the new
version is actually available from a mirror. This protocol operates
every 15 minutes (giving rise to a rounding error of up to 15
minutes).

Recent results are as follows:

2018-07-26 05:18:02 No delay
2018-07-26 13:18:02 No delay
2018-07-26 13:33:01 No delay
2018-07-26 13:48:01 No delay
2018-07-26 14:03:01 No delay
2018-07-26 14:18:02 No delay
2018-07-26 21:33:02 00:45:00 delay
2018-07-27 05:03:02 No delay
2018-07-27 13:18:02 No delay
2018-07-27 13:18:02 No delay
2018-07-27 21:34:05 No delay
2018-07-27 21:34:05 No delay
2018-07-28 05:48:02 00:30:01 delay
2018-07-28 13:18:02 No delay
2018-07-28 21:18:02 00:30:01 delay
2018-07-29 05:33:01 00:29:59 delay
2018-07-29 14:18:02 01:30:00 delay
2018-07-29 21:18:02 00:30:00 delay
2018-07-30 04:48:02 No delay
2018-07-30 13:33:01 00:44:59 delay
2018-07-30 21:48:02 01:00:00 delay
2018-07-31 05:18:02 No delay
2018-07-31 13:33:02 No delay

Note: The ClamAV update protocol uses the URL "database.clamav.net<http://database.clamav.net>",
which in turn resolves to 1 of 5 IP addresses, which in turn route to
one of many geographically separated systems. Thus, the delays may not
correspond to problems with particular paths and/or mirror instances.


P.S. I can provide a more detailed freshclam log for this if desired.


_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net<mailto:clamav-***@lists.clamav.net>
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net<mailto:clamav-***@lists.clamav.net>
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[Joel Esler (jesler)]

Thank you. We have to make adjustments very slowly to not disrupt anyone.

Cloudflare has helped us save 2 PB in the last month, delivering updates an average of 39% faster. We are seeing excellent results.

> On Aug 18, 2018, at 1:09 AM, Paul Kosinski <clamav-***@iment.com> wrote:
>
> Joel,
>
> Still lots of delays since "2018-08-11 13:18:02 No delay", but none
> quite as long as the previous batch:
>
> 2018-08-11 21:33:02 00:15:00 delay
> 2018-08-12 05:48:02 01:00:00 delay
> 2018-08-12 14:33:01 01:15:00 delay
> 2018-08-12 22:48:02 01:00:00 delay
> 2018-08-13 05:18:01 No delay
> 2018-08-13 13:18:02 No delay
> 2018-08-13 21:33:01 00:14:59 delay
> 2018-08-14 05:18:01 No delay
> 2018-08-14 13:18:02 No delay
> 2018-08-14 21:33:02 00:30:01 delay
> 2018-08-15 05:03:02 No delay
> 2018-08-15 13:48:01 00:45:00 delay
> 2018-08-15 22:03:01 No delay
> 2018-08-16 05:03:02 No delay
> 2018-08-16 14:03:02 01:00:01 delay
> 2018-08-16 21:18:01 00:14:59 delay
> 2018-08-17 06:03:01 No delay
> 2018-08-17 13:33:02 00:30:01 delay
> 2018-08-17 21:03:02 No delay
>
>
> On Thu, 16 Aug 2018 22:13:48 +0000
> "Joel Esler (jesler)" <***@cisco.com> wrote:
>
>> Paul, how are things looking from your side?
>>
>> --
>> Joel Esler
>> Sr. Manager
>> Community, Branding, and Open Source
>> Talos Group
>> http://www.talosintelligence.com
>>
>> On Aug 11, 2018, at 6:12 PM, Joel Esler (jesler)
>> <***@cisco.com<mailto:***@cisco.com>> wrote:
>>
>> I actually just made an adjustment today to see if that will resolve
>> the issues. Please keep these coming?!
>>
>> Sent from my iPad
>>
>> On Aug 11, 2018, at 2:10 PM, Paul Kosinski
>> <clamav-***@iment.com<mailto:clamav-***@iment.com>> wrote:
>>
>> Here is the latest report for ClamAV virus update mirror delays since
>> the end of July. DNS TXT vs actual file availability has gotten
>> worse! Now, over 57% of updates are DNS-announced prematurely. (Last
>> time, only about 1/3 were announced prematurely.)
>>
>> 2018-07-31 21:33:01 No delay
>> 2018-08-01 05:03:01 00:14:59 delay
>> 2018-08-01 13:18:01 No delay
>> 2018-08-01 21:03:02 00:15:00 delay
>> 2018-08-02 05:18:02 No delay
>> 2018-08-02 13:48:02 00:30:00 delay
>> 2018-08-02 21:48:01 00:15:00 delay
>> 2018-08-03 05:18:02 No delay
>> 2018-08-03 14:18:02 00:30:01 delay
>> 2018-08-03 21:48:02 00:30:01 delay
>> 2018-08-04 05:18:01 No delay
>> 2018-08-04 13:18:02 No delay
>> 2018-08-04 21:33:01 00:14:59 delay
>> 2018-08-05 05:48:02 01:00:00 delay
>> 2018-08-05 13:03:02 00:15:00 delay
>> 2018-08-05 21:03:02 00:15:00 delay
>> 2018-08-06 05:18:02 No delay
>> 2018-08-06 13:18:02 No delay
>> 2018-08-07 00:03:01 02:45:00 delay
>> 2018-08-07 05:18:02 No delay
>> 2018-08-07 13:18:02 No delay
>> 2018-08-07 21:48:01 00:29:59 delay
>> 2018-08-08 05:48:01 01:00:00 delay
>> 2018-08-08 13:48:02 00:30:01 delay
>> 2018-08-08 21:18:01 No delay
>> 2018-08-09 05:33:02 No delay
>> 2018-08-09 14:33:01 01:15:00 delay
>> 2018-08-09 22:33:01 00:44:59 delay
>> 2018-08-10 05:48:01 01:00:00 delay
>> 2018-08-10 13:48:02 00:30:00 delay
>> 2018-08-11 00:48:02 03:30:00 delay
>> 2018-08-11 05:33:02 No delay
>> 2018-08-11 13:18:02 No delay
> _______________________________________________
> clamav-users mailing list
> clamav-***@lists.clamav.net
> http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml

_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[Joel Esler (jesler)]

CC'ing your comments over to Micah. We have a heavy freshclam rewrite in the pipeline.

The amount of people using ClamAV version 0.90 and below is surprising as well. None of those versions support .diff files on the daily file. So, those versions are downloading the whole daily.cvd (sometimes hundreds of times a day) even though those versions of ClamAV won't work with the current options in daily.cvd, so they can't even start up. But Freshclam is still updating!



> On Aug 20, 2018, at 10:25 PM, Paul Kosinski <clamav-***@iment.com> wrote:
>
> It's good to save so much (5 PB) Internet traffic.
>
> What we were seeing from our end was that there were a lot of full-size
> downloads of daily.cvd that were useless because they were the old
> version rather than the new version advertised by the DNS TXT record.
>
> Besides being annoying because of lots of extra logging by freshclam,
> it kept killing off the mirror IP addresses due to the update failures,
> and thus eventually blocked all downloads.
>
> Since we already had a wrapper around freshclam to do some extra stuff
> in our environment, I decided to write the extra code to only invoke
> freshclam if the prefix of the cvd file(s) showed the correct version.
> After that, it was easy to log the delay to separate file.
>
> I guess my question at this point is: how many other users of freshclam
> are seeing the problem we had? The behavior we were seeing not only
> wasted bandwidth, it also caused semi-permanent blockage of future
> updates. Users who don't monitor their logs (like many desktop users?)
> could be far out of date with their ClamAV signatures.
>
> P.S. It shouldn't be too hard to modify freshclam itself to deal with
> this problem in a similar fashion. But I didn't want to fork a fairly
> complicated program which mainly does stuff that has nothing to do with
> this particular problem.
>
>
>
> On Mon, 20 Aug 2018 15:43:14 +0000
> "Joel Esler (jesler)" <***@cisco.com> wrote:
>
>> Thank you. We have to make adjustments very slowly to not disrupt
>> anyone.
>>
>> Cloudflare has helped us save 2 PB in the last month, delivering
>> updates an average of 39% faster. We are seeing excellent results.
>>
>>> On Aug 18, 2018, at 1:09 AM, Paul Kosinski <clamav-***@iment.com>
>>> wrote:
>>>
>>> Joel,
>>>
>>> Still lots of delays since "2018-08-11 13:18:02 No delay", but none
>>> quite as long as the previous batch:
>>>
>>> 2018-08-11 21:33:02 00:15:00 delay
>>> 2018-08-12 05:48:02 01:00:00 delay
>>> 2018-08-12 14:33:01 01:15:00 delay
>>> 2018-08-12 22:48:02 01:00:00 delay
>>> 2018-08-13 05:18:01 No delay
>>> 2018-08-13 13:18:02 No delay
>>> 2018-08-13 21:33:01 00:14:59 delay
>>> 2018-08-14 05:18:01 No delay
>>> 2018-08-14 13:18:02 No delay
>>> 2018-08-14 21:33:02 00:30:01 delay
>>> 2018-08-15 05:03:02 No delay
>>> 2018-08-15 13:48:01 00:45:00 delay
>>> 2018-08-15 22:03:01 No delay
>>> 2018-08-16 05:03:02 No delay
>>> 2018-08-16 14:03:02 01:00:01 delay
>>> 2018-08-16 21:18:01 00:14:59 delay
>>> 2018-08-17 06:03:01 No delay
>>> 2018-08-17 13:33:02 00:30:01 delay
>>> 2018-08-17 21:03:02 No delay
>>>
>>>
>>> On Thu, 16 Aug 2018 22:13:48 +0000
>>> "Joel Esler (jesler)" <***@cisco.com> wrote:
>>>
>>>> Paul, how are things looking from your side?
>
>>
> _______________________________________________
> clamav-users mailing list
> clamav-***@lists.clamav.net
> http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml

_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[G.W. Haywood]

Hi there,

On Tue, 21 Aug 2018, Joel Esler wrote:

> The amount of people using ClamAV version 0.90 and below is
> surprising as well.

That's not really surprising to me. Most of them probably don't even
know that they're running it, and those who do could easily be lying
as it's trivial to forge a User-Agent string.

Especially given what's happened in the past to users of old versions,
if there is any surprise it's that you're still serving files to them.
In my view it would be perfectly reasonable to block them. It might
even save you some money.

--

73,
Ged.
_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[Joel Esler (jesler)]

On Aug 21, 2018, at 12:32 PM, G.W. Haywood <***@jubileegroup.co.uk<mailto:***@jubileegroup.co.uk>> wrote:

Hi there,

On Tue, 21 Aug 2018, Joel Esler wrote:

The amount of people using ClamAV version 0.90 and below is
surprising as well.

That's not really surprising to me. Most of them probably don't even
know that they're running it, and those who do could easily be lying
as it's trivial to forge a User-Agent string.

Especially given what's happened in the past to users of old versions,
if there is any surprise it's that you're still serving files to them.
In my view it would be perfectly reasonable to block them. It might
even save you some money.


We have blocked people that are 0.80 and below, to see if anyone brings it up (to which, I think this list would violently react with something akin to "You are running 13 year old AV?"). No one has, publicly or privately. We'll probably proceed with a blog post stating that we're blocking everyone below the version that introduced diff'ing (0.93.3). Also rate limiting people that are attempting to download the main.cvd every 1 minute has helped.

The good news is, the top ten successful download versions (by User-Agent) are within the last 4 or 5 releases.

(0.99.4 is our largest deployed version, followed by 0.100.1, for those of you that are curious)

--
Joel Esler
Sr. Manager
Community, Branding, and Open Source
Talos Group
http://www.talosintelligence.com