[clamav-users] Partial downloads of updates

[David Rosenstrauch]

I've been having some issues over the last few weeks with freshclam
failing to download updates. It appears that it downloads the updates
the majority of the way (e.g., 95-99%) but then times out before it
finishes the download. (See example log output below.)

This may not necessarily be an issue with clamav/freshclam itself, as
the problem looks like it might be related to ipv6. (I think I was able
to get freshclam to successfully update from an ipv4 site during one
recent debugging session.) However, ipv6 definitely does appear to be
working on my machine - I do have an ipv6 address for instance, and can
successfully access http://ipv6.google.com, and all other internet
traffic appears to be working on the box. So if it's an ipv6 issue, I'm
not clear what it is. (Perhaps my router's support for ipv6 somehow
isn't complete and/or my server is misconfigured for ipv6.)

Any pointers in the right direction on how to solve this issue would be
greatly appreciated, as I'm rather stumped myself ... and my clam virus
defs remain out of date until I can get this fixed.

Thanks,

DR

---

$ sudo freshclam --verbose --debug
Retrieving http://database.clamav.net/daily-24792.cdiff
Trying to download http://database.clamav.net/daily-24792.cdiff (IP:
2400:cb00:2048:1::6810:b98a)
Downloading daily-24792.cdiff [ 97%]
nonblock_recv: recv timing out (30 secs)
WARNING: getfile: Download interrupted: Operation now in progress (IP:
2400:cb00:2048:1::6810:b98a)
WARNING: getpatch: Can't download daily-24792.cdiff from database.clamav.net
Querying daily.24792.91.0.0.2400cb0020480001000000006810b98a.ping.clamav.net
Retrieving http://database.clamav.net/daily-24792.cdiff
Trying to download http://database.clamav.net/daily-24792.cdiff (IP:
2400:cb00:2048:1::6810:bc8a)
Downloading daily-24792.cdiff [ 97%]
^C
_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[David Rosenstrauch]

Post by David Rosenstrauch
I've been having some issues over the last few weeks with freshclam
failing to download updates.  It appears that it downloads the updates
the majority of the way (e.g., 95-99%) but then times out before it
finishes the download.  (See example log output below.)
This may not necessarily be an issue with clamav/freshclam itself, as
the problem looks like it might be related to ipv6.
$ sudo freshclam --verbose --debug
Retrieving http://database.clamav.net/daily-24792.cdiff
2400:cb00:2048:1::6810:b98a)
Downloading daily-24792.cdiff [ 97%]
nonblock_recv: recv timing out (30 secs)
2400:cb00:2048:1::6810:b98a)
WARNING: getpatch: Can't download daily-24792.cdiff from
database.clamav.net
Querying
daily.24792.91.0.0.2400cb0020480001000000006810b98a.ping.clamav.net
Retrieving http://database.clamav.net/daily-24792.cdiff
2400:cb00:2048:1::6810:bc8a)
Downloading daily-24792.cdiff [ 97%]
^C

BTW, forgot to mention:

Not sure if this is relevant or not, but the above "Download
interrupted: Operation now in progress" message is what I get when I run
freshclam at the command line. When I run it via cron in the middle of
the night I get a different message. (But still a failure.)

Again ... stumped.

Thanks,

DR

---

ClamAV update process started at Mon Jul 30 02:35:01 2018
WARNING: Your ClamAV installation is OUTDATED!
WARNING: Local version: 0.100.0 Recommended version: 0.100.1
DON'T PANIC! Read https://www.clamav.net/documents/upgrading-clamav
main.cld is up to date (version: 58, sigs: 4566249, f-level: 60,
builder: sigmgr)
nonblock_recv: recv timing out (30 secs)
WARNING: getfile: Download interrupted: Inappropriate ioctl for device
(IP: 2400:cb00:2048:1::6810:b98a)
WARNING: getpatch: Can't download daily-24792.cdiff from database.clamav.net
nonblock_recv: recv timing out (30 secs)
WARNING: getfile: Download interrupted: Inappropriate ioctl for device
(IP: 2400:cb00:2048:1::6810:ba8a)
WARNING: getpatch: Can't download daily-24792.cdiff from database.clamav.net
nonblock_recv: recv timing out (30 secs)
WARNING: getfile: Download interrupted: Inappropriate ioctl for device
(IP: 2400:cb00:2048:1::6810:bd8a)
WARNING: getpatch: Can't download daily-24792.cdiff from database.clamav.net
WARNING: Incremental update failed, trying to download daily.cvd
nonblock_recv: recv timing out (30 secs)
WARNING: getfile: Download interrupted: Inappropriate ioctl for device
(IP: 2400:cb00:2048:1::6810:bc8a)
WARNING: Can't download daily.cvd from database.clamav.net
Trying again in 5 secs...
...
_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/c

[Joel Esler (jesler)]

Try the freshclam that is included with version 0.100.1 and see if you still see the error.

I've been having some issues over the last few weeks with freshclam failing to download updates. It appears that it downloads the updates the majority of the way (e.g., 95-99%) but then times out before it finishes the download. (See example log output below.)
This may not necessarily be an issue with clamav/freshclam itself, as the problem looks like it might be related to ipv6.
$ sudo freshclam --verbose --debug
Retrieving http://database.clamav.net/daily-24792.cdiff
Trying to download http://database.clamav.net/daily-24792.cdiff (IP: 2400:cb00:2048:1::6810:b98a)
Downloading daily-24792.cdiff [ 97%]
nonblock_recv: recv timing out (30 secs)
WARNING: getfile: Download interrupted: Operation now in progress (IP: 2400:cb00:2048:1::6810:b98a)
WARNING: getpatch: Can't download daily-24792.cdiff from database.clamav.net
Querying daily.24792.91.0.0.2400cb0020480001000000006810b98a.ping.clamav.net
Retrieving http://database.clamav.net/daily-24792.cdiff
Trying to download http://database.clamav.net/daily-24792.cdiff (IP: 2400:cb00:2048:1::6810:bc8a)
Downloading daily-24792.cdiff [ 97%]
^C

Not sure if this is relevant or not, but the above "Download interrupted: Operation now in progress" message is what I get when I run freshclam at the command line. When I run it via cron in the middle of the night I get a different message. (But still a failure.)
Again ... stumped.
Thanks,
DR
---
ClamAV update process started at Mon Jul 30 02:35:01 2018
WARNING: Your ClamAV installation is OUTDATED!
WARNING: Local version: 0.100.0 Recommended version: 0.100.1
DON'T PANIC! Read https://www.clamav.net/documents/upgrading-clamav
main.cld is up to date (version: 58, sigs: 4566249, f-level: 60, builder: sigmgr)
nonblock_recv: recv timing out (30 secs)
WARNING: getfile: Download interrupted: Inappropriate ioctl for device (IP: 2400:cb00:2048:1::6810:b98a)
WARNING: getpatch: Can't download daily-24792.cdiff from database.clamav.net
nonblock_recv: recv timing out (30 secs)
WARNING: getfile: Download interrupted: Inappropriate ioctl for device (IP: 2400:cb00:2048:1::6810:ba8a)
WARNING: getpatch: Can't download daily-24792.cdiff from database.clamav.net
nonblock_recv: recv timing out (30 secs)
WARNING: getfile: Download interrupted: Inappropriate ioctl for device (IP: 2400:cb00:2048:1::6810:bd8a)
WARNING: getpatch: Can't download daily-24792.cdiff from database.clamav.net
WARNING: Incremental update failed, trying to download daily.cvd
nonblock_recv: recv timing out (30 secs)
WARNING: getfile: Download interrupted: Inappropriate ioctl for device (IP: 2400:cb00:2048:1::6810:bc8a)
WARNING: Can't download daily.cvd from database.clamav.net
Trying again in 5 secs...
...
_______________________________________________
clamav-users mailing list
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
https://github.com/vrtadmin/clamav-faq
http://www.clamav.net/contact.html#ml

_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[David Rosenstrauch]

Just upgraded to 0.100.1, but still seeing the same issue.

Looks like this is going to require debugging at the network level.

Thanks,

DR

Post by Joel Esler (jesler)
Try the freshclam that is included with version 0.100.1 and see if you still see the error.

I've been having some issues over the last few weeks with freshclam failing to download updates. It appears that it downloads the updates the majority of the way (e.g., 95-99%) but then times out before it finishes the download. (See example log output below.)
This may not necessarily be an issue with clamav/freshclam itself, as the problem looks like it might be related to ipv6.

_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[G.W. Haywood]

Hi there,

Post by David Rosenstrauch
I've been having some issues over the last few weeks with freshclam
failing to download updates.

FWIW here in the UK I see no problems with IPv6 downloads.

This is the log for July 2018:

mail6:~$ >>> grep interrupted /var/log/freshclam.log
mail6:~$ >>> grep download /var/log/freshclam.log | \
sed -e 's/.*IP: \(.*\))/\1/' | sort | uniq -c
7 104.16.185.138
9 104.16.186.138
9 104.16.187.138
9 104.16.188.138
9 104.16.189.138
9 2400:cb00:2048:1::6810:b98a
9 2400:cb00:2048:1::6810:ba8a
9 2400:cb00:2048:1::6810:bb8a
9 2400:cb00:2048:1::6810:bc8a
9 2400:cb00:2048:1::6810:bd8a

As you can see there's a roughly even split between IPv4 and IPv6
downloads on this server.

Seems like you might have a comms problem. I'd be thinking of things
like traceroute, mtr, tcpdump, wireshark, etc..
--
73,
Ged.
_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[David Rosenstrauch]

Post by G.W. Haywood
Hi there,

Post by David Rosenstrauch
I've been having some issues over the last few weeks with freshclam
failing to download updates.

FWIW here in the UK I see no problems with IPv6 downloads.
mail6:~$ >>> grep interrupted /var/log/freshclam.log
mail6:~$ >>> grep download /var/log/freshclam.log | \
sed -e 's/.*IP: \(.*\))/\1/' | sort | uniq -c
7 104.16.185.138
9 104.16.186.138
9 104.16.187.138
9 104.16.188.138
9 104.16.189.138
9 2400:cb00:2048:1::6810:b98a
9 2400:cb00:2048:1::6810:ba8a
9 2400:cb00:2048:1::6810:bb8a
9 2400:cb00:2048:1::6810:bc8a
9 2400:cb00:2048:1::6810:bd8a
As you can see there's a roughly even split between IPv4 and IPv6
downloads on this server.
Seems like you might have a comms problem. I'd be thinking of things
like traceroute, mtr, tcpdump, wireshark, etc..

I finally had some time to dig into this issue, and ran wireshark on a
"freshclam" download, although I'm not sure it's helped me get any
closer to figuring out what's going on. To the best of my knowledge
(I'm definitely not an expert in networking) it looks like I'm having
some packets dropped - but again I have no idea why (or where)? I put
up a wireshark screenshot at Loading Image... which
shows a download from 2400:cb00:2048:1::6810:bd8a humming along nicely,
when all of a sudden it looks like the that remote host seems to jump
way ahead in the sequence numbering, and my server keeps re-sending
duplicate acks based on where it thinks the correct sequence number is.
In addition, "ifconfig" on my server shows 17 Rx dropped packets on
eth0. (Possibly coincidental, possibly not.)

Any idea what I might look for / where I might look from here to figure
out what's causing the issue? My server is running (an up to date) Arch
Linux installation, and is behind a Netgear WNDR3700 router. Any
suggestions appreciated!

Thanks,

DR
_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[G.W. Haywood]

Hello again,

... wireshark screenshot at http://darose.net/packets-dropped.png
which shows a download ... humming along nicely, when all of a
sudden it looks like the that remote host seems to jump way ahead in
the sequence numbering ...

Well that's obviously problematic. As you say, the sequence did get
out of shape. The packet lengths seem to be all over the place, when
I'd expect them to be more consistent for a file download.

In your OP the download which failed was daily-24792.cdiff; that same
file took under a second to download here and it came from the same IP
as it happens:

Sun Jul 29 02:18:25 2018 -> WARNING: Local version: 0.100.0 Recommended version: 0.100.1
...
Sun Jul 29 02:18:25 2018 -> daily.cvd version from DNS: 24792
Sun Jul 29 02:18:26 2018 -> Retrieving http://db.uk.clamav.net/daily-24792.cdiff
Sun Jul 29 02:18:26 2018 -> Trying to download http://db.uk.clamav.net/daily-24792.cdiff (IP: 2400:cb00:2048:1::6810:ba8a)
Sun Jul 29 02:18:26 2018 -> Downloading daily-24792.cdiff [100%]
Sun Jul 29 02:18:26 2018 -> cdiff_apply: Parsed 500 lines and executed 500 commands

I think we can suppose it's not a mirror problem, but stranger things
have happened.

There might be contention with other users or services - does this
happen at all hours of the day or is it more likely at certain times?
Joel suggested trying a later version of the package, did you do that?
I had a quick look at the changes but I saw nothing addressing this
specifically. As these aren't big files, my money's on path issues -
something like fragmentation or MTU lengths.

Your IP 2604:2000:14c4:c2da::2 indicates the ISP is Time Warner cable,
I guess anything could happen there. Do you have a way of using a
different connection, say run another box elsewhere for a few days?

Do you have a way of forcing IPv4 transport for all downloads? I see
again from your OP that you managed an IPv4 download.

... "ifconfig" on my server shows 17 Rx dropped packets on eth0.

The data lost seems likely to be more than 17 packets if the average
packet size is about what I see in your screenshot.

... behind a Netgear WNDR3700 router.

You might try a different model of router, I don't know it personally
but on a quick search I do see the odd problem report. I've had my
own issues with Netgear kit, especially Gigabit switches which will
suddenly go off the reservation and need to be rebooted. It seems to
be either just some examples of the same model, or the way that they
get hammered, I never have got to the bottom of it.

In any event it doesn't look like a ClamAV problem, so we might be
straying a little off-topic for this list.

HTH
--
73,
Ged.
_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[David Rosenstrauch]

Post by G.W. Haywood
Hello again,

... wireshark screenshot at http://darose.net/packets-dropped.png
which shows a download ... humming along nicely, when all of a
sudden it looks like the that remote host seems to jump way ahead in
the sequence numbering ...

You might try a different model of router, I don't know it personally
but on a quick search I do see the odd problem report.  I've had my
own issues with Netgear kit, especially Gigabit switches which will
suddenly go off the reservation and need to be rebooted.  It seems to
be either just some examples of the same model, or the way that they
get hammered, I never have got to the bottom of it.
In any event it doesn't look like a ClamAV problem, so we might be
straying a  little off-topic for this list.
HTH

Thanks much for the suggestions. I had thought this might be a ClamAV
issue, since this was the only download (or Internet access in general,
for that matter) that I seemed to be having a problem with. (I recently
tested downloading a several-hundred MB .iso with no issues.)

However, after researching this further, I think you're correct and it
isn't a ClamAV issue. Rather, it's an ipv6 issue, and ClamAV is the
only ipv6 site that I access on a regular basis.

Long story short, I upgraded my router to dd-wrt over the weekend, and
the issue now appears to be resolved. (I.e., I can download freshclam
updates reliably now, even from the ipv6 addresses.)

Thanks again for the help, and sorry for taking up list bandwidth.

DR

_______________________________________________
clamav-users mailing list
clamav-***@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clama