Wingate throughput

[Doog]

No CPU around 2-5%
Handles 9057
Threads 380
Processes 35

Commit Charge (K)
Total 427364
Limit 17369332
Peak 454772

Physical Memory
Total 1046516
Available 514276
System cach 592472

Kernel Memory (K)
Total 75716
Paged
51160
Nonpaged 24550


A recent timeout (WWW Server Proxy.log)
02/13/09 14:24:34 10.0.0.2 Guest 0000003473 Created:
02/13/09 14:24:34 10.0.0.2 Guest 0000003473 Requested: viewtopic.php
02/13/09 14:24:34 10.0.0.2 Guest 0000003473 Debug: [forums.qbik.com/viewtopic.php] Copy to use: SERVER
02/13/09 14:24:34 10.0.0.2 Guest 0000003473 Debug: WWW Session sending server request in thread 574
02/13/09 14:24:55 10.0.0.2 Guest 0000003473 Error: Caught socket exception in CWWWSession::HTTPProcessRequest() Connection to Remote Host timed out - terminating
02/13/09 14:24:55 10.0.0.2 Guest 0000003473 Traffic 314 761 0 0 21s
02/13/09 14:24:55 10.0.0.2 Guest 0000003473 Terminated exit code 2

(Kaspersky AV for Wingate.log)
02/13/09 14:24:33 0.0.0.0 <system> 0000003632 Debug: Obtaining lock in InterfaceAdd
02/13/09 14:24:33 0.0.0.0 <system> 0000003632 [010EA620] Interface Created (2)
02/13/09 14:24:33 0.0.0.0 <system> 0000003632 Debug: Releasing lock in InterfaceAdd
02/13/09 14:24:33 0.0.0.0 <system> 0000003632 Debug: Obtaining lock in destructor
02/13/09 14:24:33 0.0.0.0 <system> 0000003632 Debug: Releasing lock in destructor

[Nev]

The KAV log doesn't reveal anything and there is nothing in Quarantine. The KAV log has only recorded updates so I have now enabled debug logging. Interestingly, the problem appears much more acute on pages with content that is more than a few pictures and text. For example, .pdf files mostly fail, eBay and the ISAPI.DLL URL is practically unusable and I get 500 errors constantly when going through Wingate but directly it works fine.

help!!!

Guys when I see this one approach is to delete the History file, you never know it could be problematic // stop engine delete history // start engine.

Failing that, backup the Wingate registry, de-activate the licences and fully un-install Wingate - totally.

Re-install on trial licences, Wingate + KAV and try the resource that is currently timing out [quite ok here by the way via Wingate - KAV though].

Next if there is any improvement save the existing registry and import the prior one and try again in an attempt to locate the culprit.

[adrien]

Hi

Sounds like there's something else going on. That connection timeout makes me think there's some network issue.

Did you leave AVG off? Also, is this with the Cisco VPN connected?

[Doog]

Over the weekend I deinstalled Wingate, removed all the network cards and then reinstalled almost everything. At this point in time it appears to be working fine. However, I have not been able to reinstall KAV - it says that "this product is not registered with Kaspersky Labs". I therefore presume that KAV isn't operatrional.

[adrien]

normally you only get that message if the WinGate engine hasn't been signed.

All our released builds are signed to work with KAV.

Where did you get the build to install?

[Doog]

This was downloaded from qbik.com. I had this problem last year when I upgraded the hardware and I had to download a new .KEY file. I have tried the same procedure but to no avail. If I put the .KEY file in C:\Program Files\WinGate\Plugins\Kaspersky AntiVirus\Downloads or C:\Program Files\WinGate\Plugins\Kaspersky AntiVirus\ I get the following messages:

"The license for Kaspersky AntiVirus for WinGate has expired or is invalid. Kaspersky AntiVirus for WinGate functionality will not be available"
"Unable to obtain interface for 'Kaspersky AntiVirus for WinGate"

I have followed the instruction on a previous help desk ticket GAO-88130 - it seems that the license file just doesn't make any difference. If I delete it I get the same errors.

[adrien]

yep, that particular error message isn't related to our license files.

The Kaspersky Labs SDK which we use checks the host exe is signed, and checks that signature with a kind of certificate file which ships with Kaspersky AV for WinGate - called keyid.dat which lives in the Wingate folder (same folder as wingate.exe).

Is this file there?

Adrien

[Doog]

I have put the file in the following directories.

C:\Program Files\WinGate
C:\Program Files\WinGate\Plugins\Kaspersky AntiVirus
C:\Program Files\WinGate\Plugins\Kaspersky AntiVirus\Downloads

There is only one file present at any one time. Each time I reboot the gateway but get the same results.

[adrien]

I'd be happy to take a look with remote desktop or VNC to try to get to the bottom of this if you'd like

Regards

Adrien

[Doog]

Happy to do so, what do I do?

[adrien]

I just sent you an email with some instructions - let me know if you don't receive it.

Cheers

Adrien

[Doog]

Unfortunately I am not near the Wingate PC until tomorrow but I'll look at it then.

Thanks

[Doog]

Thanks Adrien, the KAV is now working again. My throughput has now dropped back to around 4.9Mbs from 10-11Mbs. It is very consistent that when I enable the KAV plugin the speed drops be around 50%. I have tried to see if the override would shed any further light on the situation but that seems not to work, KAV keeps scanning the file, perhaps I have entered the information incorrectly??

Parsed out override (www.ozspeedtest.com) vs original (www.ozspeedtest.com)
Override does not match site so page will be scanned

[adrien]

if you take a look in the history, you might find the actual file being downloaded as the test is actually at a different site, so whitelisting the speedtest site won't catch it.

Regards

Adrien

[adrien]

Hi

I did some tests of my own here using www.speedtest.net. As I mentioned before, it comes down to how the result is calculated. This can have a huge effect on the results.

These are my results all in Kbps

Direct connection (no proxy)
Down Up
7249 18165
6957 19517
7979 19242
6353 18420
7312 17253
9887 19091
9062 17117
5708 19873
8324 18628
8971 18955

Through Proxy + KAV
13095 16108
11516 17525
9274 16716
14224 17924
11501 17087
12195 16065

yes, you are seeing this right. results for going through proxy + KAV are FASTER than not.

Now we know of course they aren't actually faster - that's actually impossible, but the numbers this speedtest site is spitting back are bigger, so why is that?

Well if you look at the rate meter during the test, it actually was sitting around 5500 - 6500. When the download got to 75%, the number shot up to end the day at usually around 12000.

5500 - 6500 is about 75% of the values shown with no proxy. So, as I said, when passing through KAV, we only send 75% back through to the client, so this is expected.

What this shows, is that

a) Speedtest.net does not calculate throughput on an overall amount of data / amount of time basis. It seems to have some sort of averaging window.
b) results are quite variable. This is to be expected, since I am competing with every other packet on the path between our site and the target speedtest server.

so, All I can do is reiterate. If you want a meaningful throughput figure you have to eliminate a bunch of distortion factors.

1. You need to eliminate variability due to competing with other internet users. This means you can't use internet-based speed tests. You have to hit a server on your local LAN.
2. You need to be careful about how the throughput is calculated. If your test client is showing instantaneous throughput (e.g. averaged over a short averaging window) then you will see reduced throughput due to the 75% factor up until the file is scanned, then a very high throughput when the file is delivered to you at LAN speed.
3. You may need to take into account TCP ramp-up. Modern OSes have a start-up and speedup algorithm used for TCP. It modifies TCP window sizes to increase throughput during the course of the connection. Unfortunately you can't use anything other than TCP to test KAV, but I believe there are registry settings in some OSes to turn this off.

Regards

Adrien

[adrien]

p.s. this is using a WinGate machine on a P4 3GHz with 1GB RAM on 2k server. No http caching on the proxy.

[Doog]

I agree that there may be differences in the test methodologies used to indicate speed. Please be assured that I am not trying to discredit Wingate as a product because I think what it does is great.

I decided to use a speed test to narrow down where my problem lay. Up until last year I ran Wingate on a Pentium III with 256Mb of memory. That PC finally gave up so I bought another, slightly more powerful PC a Pentium IV with 1Gb of RAM. Over the same period I notice a certain decline in performance particularly when downloading files. The most noticeable problem was browsing and downloading stalling and failing. Either by coincidence or not when I turned KAV off, the files did download OK and browsing seemed to return to normal. This may have been a ‘red herring‘. Eventually, it seems that wiping and reloading Wingate has improved the situation.

However, with or without a speed test, KAV has a throughput impact. When I browse the Internet or download files with KAV enabled, I can see the difference in throughput. This manifests itself in two ways. First, as you have pointed out the speed is quicker for the first 75% and then there is a noticeable pause before the final 25% is downloaded. Second, the overall throughput is lower. This is simple; it takes longer to download a file through Wingate with KAV enabled. Throughput is a measure of how long it takes to download (or upload) information, therefore, simplistically, the longer it takes the slower it is.

This may be affected by the power of the Wingate PC, I get that, the process downloads data, scans it and then passes it on to the client. Therefore, the quicker it can do the middle bit the quicker it completes. Perhaps from you tests there needs to be a 3Ghz PC to process this without noticeably affecting performance. It would be interesting to see the variation of throughput from other users or am I just unlucky.

[adrien]

Hi

I wouldn't have thought the difference between a P4 2GHz and a P4 3GHz would have been that great, so I'm a bit confused about the cause of the slow-down. Our test server here also had only 1GB RAM, but was running 2k server rather than your XP Pro.

One area where we see quite different results with the speed-test. You say your first 75% is faster, then there''s a noticeable pause before you get the rest. In my testing the first 75% was slower, then no pause and the rest blitzed down.

The pause between hitting 75% (when we have the whole file, so can start the scan) and the start of sending the rest, is the time it takes to scan the file. So I guess this does mean your KAV is CPU-bound at this point. I agree any extra delay introduced in the process of getting a file reduces effective throughput, however at any one time you should find with multiple files coming down, you should be able to get the max throughput of 75% of your line speed. So throughput in this sense is misleading.

I wonder if your system (being tuned for a desktop os) starts using disk swap-space before a server OS would. You may get better results if you tune the PC a bit, e.g. go to

My Computer -> properties -> Advanced -> Performance Settings -> Advanced, and select "Background services" under processor scheduling, and perhaps System cache for Memory usage.

Cheers

Adrien

[adrien]

another thing that could be useful would be to use perfmon to log various metrics, e.g. system memory usage, swapping, CPU etc whilst you are doing these tests.

Also is hyperthreading enabled on that machine?

Cheers

Adrien

[Doog]

Adrien, I have sent some data via email, apologies I don't know how to attach it to a 'post'. I set the Wingate PC as you suggested which actually made the performance slightly worse. What I can confirm is that KAV carries out a significant disk write followed by some intensive processor activity (see graph I emailed you).

In terms of the '75%' delay, with a 9Mb file download this takes several seconds, probably 3+ seconds but hard to time accurately. This seems to correspond to the time that the disk is being written to on the Wingate PC.

In terms of Hyperthreading, I beleive the PC is capable but when I list processors in the systems tab of control panel it only lists one processo, I therefore assume it is disabled or unavailable.

[adrien]

Hi

thanks for sending those results through - they are very interesting. Is the disk write perf counter only for the kavss.exe processes or for the system as a whole? I would normally expect that this is WinGate writing the file that KAV will scan. KAV shouldn't need to write to disk to scan unless it is unpacking archives.

It's odd that it peaks like that, maybe the OS disk cache is being flushed. Taking 3s to flush a 9MB file to disk isn't very impressive. Is the disk highly fragmented or otherwise very slow?

It also looks like the actual scan of the file is taking about 9s when you are running the client on the WinGate machine (KAV competes with the client app for CPU) and slightly less time when the client is remote at about 7s.

I'm a bit confused at the amount of CPU used before KAV starts scanning in your 9MB-KAV-DIRECT case. I presume this is the client? It's possible to view CPU / mem / disk usage on a per-process basis in perfmon (choose the process object, then select which process in the RHS then select the per-process counter) if you want to spend any more time on this seeing which process is actually chewing the disk or CPU. Based on comparison between 1 and 2, I'd say it's the client (browser) chewing CPU at the start, then KAV follows on.

As for those performance settings, I would have expected the first one to improve things, but not sure about the second (memory). Did you try just the first? I would also try turning on hyperthreading if your BIOS supports it, I'm pretty sure all P4s support it.

[Doog]

Thanks. I will do some more testing, although probably later in the week. I have some work to do! II set the PC to 'Background services' under processor scheduling, and System cache for Memory usage. I will try and set 'hyperthreading' but I have no idea how to enable this. It is not an available option on the bios and I don't really want to upgrade the bios unnecessarily.

The disks are 100% defragged - I do this nightly. It also does not appear to be sluggish reading and writing from disk - the PC is fairly 'unbloated'.