Wingate throughput

[Doog]

I have been using Wingate for some time (since '02). I recently noticed that the my PC speed started to slow. I suspected the usual spyware, etc. I upgraded the PC because the old hard disk was on its way out anyway. However, when I dug a bit deeper I noticed some strange performace effects when using Wingate. I have a PC that runs exclusively Wingate (well + a printer) with two network cards installed. It connects through a netgear router to the Internet. I have tested the speed with Wingate loaded, with Wingate loaded with KAV running and without Wingate loaded. The results are as follows:

Line speed Download Speed
Wingate and KAV 3.268 0.398
Wingate no KAV 6.25 0.762
No Wingate (browser set not to use proxy) 10.008 1.224
No Wingate (Wingate unloaded) 10.352 1.262
No Wingate (Wingate unloaded and internal network card disabled) 10.456 1.276
(These are the averages of five tests each).

I have turned off all logging and all non-used services. The PC is a 2Ghz Intel Pentium 4 with 1Gb RAM running Windows XP Professional. The only other software on the PC - Cisco VPN, HP Laserjet control panel, AVG and Internet explorer. (it boots in 15 seconds!!!)
I understand that there should be a slight performance impact of using a proxy and virus checking, however, should I expect it to be this much?

Comments and sugestions welcome.

[vasilis]

Please try to unistall AVG and make the tests again, but have KAV for WG installed.

[Doog]

I have effectively tried this. The test are as follows:

Test one:
Both Wingate and KAV, browser set to use Wingate proxy Line speed=3.268 Download speed=0.398

Test two:
Both Wingate but no KAV, browser set to use Wingate proxy Line speed=6.25 Download speed=0.762

Test three:
No Wingate (browser set to bypass the proxy - in this case Wingate is doing nothing to the connection) Line speed=10.008 Download speed=1.224

Test four:
No Wingate (browser set to bypass the proxy with Wingate completely unloaded from memory) Line speed=10.352 Download speed=1.262

Test five:
Same as test four but the internal network card is disabled. Line speed=10.456 Download speed=1.276

[alyork]

Funny you should mention the apparent speed impact with Wingate installed.

I've only upgraded 1 of our servers to 6.5 and now see a 30% decrease in internet speed.

I set up another computer that uses the same interface with no wingate and its running at the same speed that wingate server used to run at.

I thought it was the internet connection and had the telco check it out, replace cables, modem etc. Managed to increase the overall internet speed but the wingate server still run 30% slower than the non-wingate machine and still slower than before the 6.5 upgrade.

Not 100% sure of course that a Microsoft upgrade or something else is not at fault. However, we are running Windows 2000 Server and it may be something like all the extra code that's be added to Wingate for Vista.

[adrien]

I'll get the guys in the lab to check this.

It shouldn't be any slower than 6.2.2.

Regards

Adrien

[adrien]

p.s. is this all machines slower through WinGate, or is just the WinGate machine itself affected?

Adrien

[Doog]

I can confirm that the speed from a client machine is the same as the test from the server with KAV loaded.

I did upgrade from the previous version to the latest over the weekend because I thought that it might be related to the previous version. My main question is what performance should I expect?

[adrien]

One other thing, how are you measuring speed, and is the traffic going through AntiVirus (e.g. KAV on WinGate)?

Depending on the CPU / memory of the machine, KAV can have quite a performance effect.

I'll get the guys in the test lab to look at this today.

Regards

Adrien

[Doog]

I can confirm that Wingate and the KAV plugin are used when the tests are set to use the proxy. The test used is at the following site:
http://www.ozspeedtest.com/bandwidth/

The specific test used is:

OptusNet Mirror

3 MB Test

[adrien]

Can you try without KAV?

KAV has to accumulate the entire file before it can scan it. If you say don't have drip-feeding turned on in the WWW proxy, then it can greatly affect reported speeds, since you won't receive any data until the entire file has been downloaded, scanned and then sent to the client.

We've been testing with that site, and we aren't seeing the slow-down you are reporting.

Also, what sort of machine is this on? CPU/mem?

Regards

Adrien

[adrien]

sorry - I see you already posted that.

AVG could still be having an impact. E.g if it is scanning the temp folders or any WinGate folder.

Can you disable or uninstall AVG completely and try that?

Adrien

[adrien]

alyork

when you have the browser on the WinGate machine configured to use the local proxy, any data coming from the internet to the browser goes up the local TCP/IP stack to the proxy, then back down the stack on the localhost interface to the protocol driver, then back up to the browser. That's 3 times across most of the TCP/IP stack.

For a client behind WinGate via proxy, the data goes up the stack to the proxy, then back down the stack and out the LAN card. That's only 2 traversals.

For a client behind WinGate using NAT, it doesn't even go all the way up the stack before WinGate sends it back out to the client. So this is the quickest in terms of throughput.

So using proxy for browsing on the WinGate machine is 1 more stack traversal than for non-wingate machines, and it does have a performance impact, esp on single or dual CPU machines.

What this means is we do expect browsing via WinGate from the WinGate machine itself to be slower than browsing from a client.

As for slowdown since upgrading to 6.5, we did make some changes in the ENS driver about handling local connections (i.e. connections made by the WinGate machine itself), I'll see if there is anywhere we may have introduced a big speed penalty.

[Doog]

OK. I have turned off KAV in the above tests that is the scenario in test two: (Wingate but no KAV, browser set to use Wingate proxy Line speed=6.25 Download speed=0.762)

The PC is a Pentium 4 running at 2Ghz with 1Gb RAM. I originally thought that the old PC (700Mhz) was the core reason for the speed reduction but, although I have no test results to hand (I could with a few hours of work), this does not seem to have made any difference. I get very similar performance from the client machines going through WIngate.

I'll try and upload the results for you.

Wingate client with KAV 4.196 0.5302
Wingate client KAV disabled 11.114 1.358

[Doog]

Adrien

I cannot upload the results of the throughput tests. I get the following error:

Could not upload attachment to ./files/1408_0f4ad7f131ec3e821386b4cb0771b9ad.

[adrien]

Hi

What about with no AVG?

WinGate spools files to temp folder for caching etc etc. If those files are rescanned by AVG every time WinGate writes another chunk, it will have a marked effect on performance. if AVG is intercepting traffic and scanning it as well that will also have a marked effect.

The only way to be sure is to completely remove AVG (I'm not talking about KAV here).

Adrien

[Doog]

Sorry didn't read your response properly. OK I have removed AVG from the Wingate PC and this has had a dramatic effect. With AVG removed the results are as follows:


Wingate client with KAV 3.984 0.486
Wingate client KAV disabled 10.916 1.334
Wingate PC with KAV 4.348 0.534
Wingate PC KAV disabled 12.252 1.496

But KAV still has a significant effect on performance.

[alyork]

p.s. is this all machines slower through WinGate, or is just the WinGate machine itself affected?

Adrien

In my case its both the wingate server and all the pc's going through it.

[adrien]

Hi Doog

Is drip-feeding enabled in the WWW proxy plugins tab?

Regards

Due to timing, if it's off, that would make your tests appear slower. I'd be tempted to try the bigger files, and see if it's still such a big difference as well.

Adrien

[Doog]

Drip feed tests. These are all from the same client machine. I have changed all the drip feed settings and run the tests using the 3Mb file for consistency. The last test was with a 9Mb file. There does not seem to be any huge difference.

Wingate client PC with KAV 3MB test with 128Kb drip feed 4.184 0.51
Wingate client PC with KAV 3MB test no drip feed 3.864 0.47
Wingate client PC with KAV 3MB test with 4096Kb drip feed 3.752 0.458
Wingate client PC with KAV 3MB test with 3072Kb drip feed 3.91 0.476
Wingate client PC with KAV 3MB test with 2048Kb drip feed 4.48 0.546
Wingate client PC with KAV 3MB test with 1024Kb drip feed 4.268 0.52
Wingate client PC with KAV 3MB test with 1000Kb drip feed 4.472 0.546
Wingate client PC with KAV 9MB test with 128Kb drip feed 4.368 0.536

[Doog]

I am now only testing from the client machine because I don't need to bypass the proxy.

[adrien]

regarding the test file size, the way drip-feeding works is that

a) we only send 75% of the file as it's being received.
b) whilst scanning, on a timer we send a part of the remainder of the file.
c) when scanning is complete we send the final amount.

So depending on overall length of time it takes to download the test file, (in your case only 1 or 2 s), some timers might not get a chance to kick in, which would skew results, since the end result is basically extrapolated over a short period of time.

Ideally something that takes about a minute to download will give you a better idea of real throughput.

Regards

Adrien

[Doog]

Thanks, I'll see what I can do.

[Doog]

OK. Wasn't sure how to get a longer test but I donwloaded a 97.8Mb file from a distant site so the accurancy may not be consistent or good enough. The results are as follows:


Client PC through Wingate (KAV enabled) Took 816.00 seconds therefore average download rate 125,674.92
Wingate PC direct (no proxy) Took 610.00 seconds therefore average download rate 168,115.96
Wingate PC through proxy (KAV enabled) Took 722.00 seconds therefore average download rate 142,037.03
Wingate PC through proxy (KAV disabled) Took 583.00 seconds therefore average download rate 175,901.77

I have calculated the download rate by dividing the file siize in bytes by the time in seconds (I think that's the right way to do it).

[adrien]

OK

It's often difficult to get reliable repeatability with distant sites, since traffic speeds can vary greatly depending on various loads at your ISP, or upstream, or even load on the remote server.

In general the most reliable way to get throughput data is if you are hosting a file locally, with nothing else loading up that LAN.

However, it does seem that there is a penalty to be had for running KAV, which isn't entirely unexpected. In the end, the main difference between running KAV or not (apart from the time it takes to actually scan the file - this is extra time over not having to scan it), is the disk spooling or CPU loading. A 2GHz P3 isn't a rocket, but it's not a slug either. You may find if you need higher performance that you can get this by upgrading that machine's CPU, RAM or Disk subsystem.

You may also find if you ran this test twice simultaneously, that the total throughput would be more than you are seeing in the single case.

Cheers

Adrien

[Doog]

I have come to the conclusion that the throughput is being choked by KAV. I did some more testing specifically with KAV and noticed that some files that are downloaded are incomplete and I am unsure whether this is a set-up issue or something else. Here is my example. I download the following file from Bigpond. (I got the URL from Wingate)

http://192.148.123.45/wm9.streaming.tel ... iba_2M.wmv

If KAV is enabled I never get the full file. If I disable KAV it comes through OK. By changing the drip feed setting I get more and more of the file but never the complete file - the browser just says 'buffering'. This is worse from the client machines. Do you think I have a set up issue or is this the expected outcome?

[logan]

When drip feeding is enabled, the first 75% of the file is passed on to the client as it's downloaded by the proxy. The last 25% is then held back so that plugins can scan the file and decide whether it should be passed to the client, or dropped.

What's probably happening is you are streaming fine up until the 75% mark, at which point the stream stalls and trys to buffer while WinGate is downloading the last 25%. When WinGate is done downloading the file, and the plugins have OK'd it, the client should get the last 25% in a quick lump.

Can you try waiting a while and see if the client gets that last 25% of the file or not?

[Doog]

I have waited a very long time for one document (24 hours) I actually went on a bus and picked up a hard copy. There is definitely something wrong. Here is an example:

02/12/09 17:12:42 10.0.0.4 Guest 0000003199 Created:
02/12/09 17:12:42 10.0.0.4 Guest 0000003199 Requested: http://www.challenger.com.au/PDS/HMT_RETAIL_PDS.pdf
02/12/09 17:12:42 10.0.0.4 Guest 0000003199 Debug: [www.challenger.com.au/PDS/HMT_RETAIL_PDS.pdf] Copy to use: SERVER
02/12/09 17:12:42 10.0.0.4 Guest 0000003199 Debug: WWW Session sending server request in thread e5c
02/12/09 17:12:42 10.0.0.4 Guest 0000003199 Debug: Server response contains 2502 bytes of resource data
02/12/09 17:12:42 10.0.0.4 Guest 0000003199 Debug: WWW Session processing HTTP response in thread e5c - response code 200
02/12/09 17:12:42 10.0.0.4 Guest 0000003199 Debug: Drip-feeding resource to client
02/12/09 17:16:42 10.0.0.4 Guest 0000003199 Traffic 576242 660 548 576243 240s
02/12/09 17:16:42 10.0.0.4 Guest 0000003199 Debug: Server closed connection in thread e5c
02/12/09 17:16:42 10.0.0.4 Guest 0000003199 Traffic 520816 660 548 576243 0s
02/12/09 17:16:42 10.0.0.4 Guest 0000003199 Terminated exit code 2


I never get the page in the browser when I go through Wingate. I can by reconfiguring the browser on the Wingate PC and setting to bypass the proxy.

[adrien]

that exit code 2 means the session timed out due to inactivity.

So it's nothing to do with KAV I don't think.

Were there any errors logged in the KAV log file? Or is there anything showing in the quarantine?

Adrien

[Doog]

You are probably right and I will do some more testing. However, my symptoms are that browsing is so incredibly slow, pages can take tens of minutes to load and they often time out. That is why I originally started looking at the problem. I am fairly convinced that is is something to do with Wingate and probably KAV. The reason I say this is that when I connect directly (and bypass Wingate) everything works perfectly and it is very very quick. When I go through Wingate it is noticably slower and when I enable KAV it crawls and often times out. I just cannot figure out why this is the case. I am quite happy to accept that there is something else wrong but the sypmtoms only appear through Wingate.

The KAV log doesn't reveal anything and there is nothing in Quarantine. The KAV log has only recorded updates so I have now enabled debug logging.

Interestingly, the problem appears much more acute on pages with content that is more than a few pictures and text. For example, .pdf files mostly fail, eBay and the ISAPI.DLL URL is practically unusable and I get 500 errors constantly when going through Wingate but directly it works fine.

help!!!

[adrien]

What does the CPU and memory usage look like on the gateway when this is happening? does the HD thrash or anything like that?

Regards

Adrien