Wingate hogs Memory downloading large files

[pgr]

Hi,

I've been having a problem with Wingate 6.2.0 - perhaps a misconfiguration on my behalf.

When users download large files, like 600Mb CD images, Wingate seems to put all of that file in RAM while downloading, which almost brings my server to a halt (it's an old machine with only 384Mb RAM).

I notice the Internet is slow, I go up to the server and open Gatekeeper and find out somebody is downloading a large file. The server is swapping like crazy and the Wingate process shows something like 600Mb virtual memory. Eventually the download finishes and everything goes back to normal.

I don't know if this is relevant, but in WWW Proxy Server properties I have "Dripfeed data to the client if file is bigger than 200kb". I have a single Plugin: Puresight.

Is there any way to avoid this problem? It's becoming a nuisance since large downloads are always getting more common and I wish I didn't have to upgrade the server: Wingate runs perfectly on that old machine, except for this problem. It seems there could be better ways of handling large files.

Thanks in advance for any help!

[pgr]

...

am I the only one that has this sort of problem?...

Is there any way to start understanding what could be wrong?

Is this a bug?...

thanks

[alyork]

There is something about Kaspersky downloading a whole file before scanning and releasing it. There is also something under www.proxy.server plugins called dripfeed that may help if its checked on. My understanding is that dripfeed passes most of the file on except the last 25% until Kaspersky has proofed the file.

I don't know where Kaspersky keeps the file until after its scanned and I don't know why dripfeed is off by default.

Anyway, its just a guess on my part that may or may not help solve your problem.

- Al

[pgr]

I don't have Kaspersky installed, my only my only plugin is Puresight... anyway, I have dripfeed turned on in WWW Proxy Server, so I don't think that is the problem...

[rboynton]

Hi PGR,

Well, you've done well to allow the old box to keep running this long. As you know, 384 RAM is puny in today's world. When an application needs resources, it grabs all available RAM then swaps the rest. Your O/S takes a good hunk of that 384, then WG takes some. If WG needs to process a large file, it takes even more during processing. This is why you're running out of resources. You probably also had messages come up like "windows is running low on virtual memory", or the like.

As inexpensive as boxes are these days, you might want to replace the old fellow and give him a retirement ceremony commensurate with his long faithful service. Trying to upgrade the RAM on old boxes is sometimes too expensive for what you're getting, but that's your decision.

[adrien]

Hi

When a large file is being downloaded, have you noticed whether WinGate memory usage grows whilst it is being downloaded, or only when it is being scanned (at the end)?

Drip-feeding settings shouldn't affect this - they only affect whether or not any of the received data (whether accumulated in memory up to 200kb or spooled to disk) is sent through to the client as it is coming in.

Regards

Adrien

[pgr]

When a large file is being downloaded, have you noticed whether WinGate memory usage grows whilst it is being downloaded, or only when it is being scanned (at the end)?

It's hard to say, my attention is only drawn to it when the server starts slowing down and I go there to see what's going on. But I do know that when I get there and see a large amount of Virtual Memory for Wingate.exe, it doesn't seem to be growing. It just stays roughly the same for about 10 minutes (with disk swapping all the time) as it is doing something to the file. I would say that it probably has already finished downloading, and is doing some final operation on it.

When you say "being scanned" does that mean Anti-virus? I don't have Kaspersky. Does Puresight also scan through the whole 600 Mb .rar file?

[adrien]

PureSight is only supposed to scan text/* mime-types received from a server.

I'll need to get this tested in the lab - do you have a particular URL we can test with, or does it seem to do it with many?

[pgr]

PureSight is only supposed to scan text/* mime-types received from a server.

I'll need to get this tested in the lab - do you have a particular URL we can test with, or does it seem to do it with many?

It happens with many URL's. The ones I've seen where movies in .rar compressed format coming from sites like rapidshare and the likes.

I also experienced problems like the one recently described here: http://forums.qbik.com/viewtopic.php?f=12&t=39421. Large files that I cannot download from a client, but I can download them if I go to the server and download directly from there (i.e. not through Wingate). It could be exactly the same issue, only if the file is really very big the whole thing never finishes.

Thanks for your attention!

[pgr]

I watched a large download closely, looking simultaneously at the client PC (Vista with Firefox 3.0) and at the Wingate server (Win XP), seen through Remote Desktop.

The URL is
http://download3.vmware.com/software/vm ... 101586.exe

and it is coming through WWW Proxy Server Service with Puresight plugin active. No Kaspersky...

1- The download progresses fine, with very good speeds, up to 402Mb. The total file size is 534Mb, so this would be around 75%... at this point, it slows down significantly.

2- On the server side, at this exact time, the virtual memory of Wingate.exe jumps up by the amount of 534Mb. This makes the server slow and irresponsive. Wingate is doing some processing, but not much (under 7% CPU).

3- On the client side, a few more Mb come in, download is up to 430MB, then it stalls completely. I try opening a webpage from the browser, but wingate is not handling requests anymore and I get timeouts.

4- After about 10 minutes in this state, it seems Wingate (or Puresight?) finishes it's work and the VM drops back down abruptly. The server suddenly starts handling the web page request I had given it before. CPU processing goes up for a minute or two as I see Wingate is "Purging Web Cache" and "Rebuilding Index". At this point, I remember to check and my wingate cache is limited to 400Mb...

5- On the client side, the download is still stalled and nothing happens. After a few minutes, I quit it.

SECOND ROUND
------------
I push up my Wingate cache limit to 2048Mb and try the same download again.

1- everything goes ok until 75% of the download. At this point the VM jumps up again abruptly, but this time, the download seems to continue, slower, but not stalling...

2- The download speed keeps getting slower and slower, but never stops dripping. On the server, I see the VM stays high until the download finishes... it started at 650Kb/s (sustained during the first 402Mb) and ended at under 5Kb/s!

3- Wingate keeps serving web pages to everyone during this time.

4- When after a LONG time the download completes, Wingate.exe Virtual Memory immediately drops back down.

I hope this helps. I would really appreciate a soltution to this problem! Even though it seems to be better now, with a higher cache, Wingate really shouldn't be handling large files like this, and Puresight wasn't even supposed to be looking at .exe files...


Thanks.

[pgr]

After upgrading Puresight plugin to version 3.0, this problem is completely solved... thank you Qbik, this was a very needed fix!