Very Large File Downloads Incomplete

[alyork]

For several weeks now, a client has been trying to download the trial version of Autocad 2009 from AutoDesk.

The first problem was that the browser was sitting on "loading" for hours and doing nothing else. I resolved that one by clearing out the cache.

Ok, so the download now runs for several hour, as we are talking about a 1.2 gig file, and then says completed. However the resulting .exe file is always smaller than the source file and will not run. 4 or 5 attempts produced the same bad results.

I then tried the download on a computer that was directly connected to the internet outside of the Wingate protected area. The download worked fine with a valid executable as a result.

Dripfeed is turned on in Wingate.

There seems to be problems with Wingate and downloading files. Adding every URL we think we want to download from into bypasses and ignored site lists seem to be bit silly. And we don't even know if that would work.

Is there something more intelligent we can do to to resolve the large file download problems?

Thanks - Al

[pgr]

Hi,

I'm having a similar problem which I described here: http://forums.qbik.com/viewtopic.php?f=12&t=39414&p=32001#p32001.

Perhaps you can try to observe if you have similar symptoms: huge memory consumption at the end of the download... also, what plugins do you have in Wingate (Kaspersky, Puresight)?

[logan]

It would also pay to check the quarantine in WinGate. If a download is scanned and deemed infected by the Kaspersky AntiVirus plugin, WinGate will cancel the download and not pass the remaining %25 percent of the file to the client. This would result in downloaded file on the client computer becoming corrupt, and smaller than expected.

[adrien]

Hi

It's possible that due to the size of the file, it is basically impossible to scan it, and a scan error is leaving the downloaded file corrupted.

Whilst the file is being drip-fed to the client, we never send the whole file until scanning is complete. If scanning fails, depending on Kaspersky AV for WinGate settings, we will either pass the file on, or quarantine it (in which case a final chunk is sent to make the file invalid).

So check the quarantine as Logan suggested, also if you have a text editor that can open such a large file without loading the complete thing into memory, check the end of the file for scanning warning / error messages.

[alyork]

I think I'm going to start screaming!!

Self corrupting scanning? What next?

First the computers that attempted the downloads all have 4 gig ram.

And yes, the file were corrupted thanks to ?? Wigate?? Kaspersky?? Puresite?? something else?? or a combination of them all??

"07/18/08 08:48:02 127.0.0.1 guest 0000006068 Kaspersky AntiVirus 2.0 for WinGate has quarantined http://trialdownload.autodesk.com/enu/a ... _32bit.exe for guest because it is corrupted"

No mention of any infection.

The computer we finally got a good download on the first time we tried was running AVG antivirus/antispyware and was happy with downloaded file.

[adrien]

when an HTTP proxy detects that a file it is passing to a client is rejected by AV (for whatever reason - in your case looks like KAV is considering it to be "corrupted"), then it has few options.

depending on what has been sent and how, simply closing the connection may not cause the browser to see any problem with the file.

Consider the case where you're actually downloading an exe that actually contains a virus. the conundrum is that if we don't pass anything through to the client while we are downloading and scanning the file, then the client doesn't see any activity for a while, and the user will typically quickly give up waiting and hit refresh quite a few times before giving up and calling their sys admin.

So, we added drip-feeding, to keep the human attached to the browser happy that something was happening.

The downside of this, is that it's impossible to guarantee that whatever is sent through is actually safe. For this reason we wish to deprecate drip-feeding, and I wrote a draft internet specification for a resolution to this problem which has been largely ignored by the HTTP standards development community. I guess they don't consider it to be a problem.

https://datatracker.ietf.org/drafts/draft-decroy-http-progress/

anyway. So lets presume we are stuck with dripfeeding, so the proxy has been feeding a file (75% of it) through to the client as it gets received, and further smaller chunks of it whilst it is being scanned so the client doesn't time out due to long scanning periods (scanning large archives can take a while).

Then we discover there is a virus.

If you just close the connection, if there was no Content-Length header received with the response from the server, and we weren't sending it chunked, then the client will think it got the whole file.

If we were sending it chunked, or there was a Content-Length header, (more likely), then the client will consider the download aborted, and may automatically retry.

Alternatively some clients will simply leave the file where it was being downloaded to, and you have a file on disk you may double click on to try and run.

So the decision was made to "poison" the file by sending something on the end, which also identifies it (if you look) as a virus.

In your case, it appears that the problem is that KAV is deeming the file corrupted, and quarantining it, this is causing the other problems. You can change settings in KAV as to what to do for corrupted files - e.g. pass it through. That will solve this problem.

we are currently working with Kaspersky Labs getting them to develop a streaming interface to their scanning engine so that we can identify a virus in stream, without having to accumulate the entire file before scanning it.