WG 6 stops servicing www requests

[saubrey]

I have WG 6 984. After 4 days of running, WG 6's www proxy just stopped servicing requests. FTP proxy, socks proxy, etc. all continue to work correctly, but WWW proxy stopped processing requests. The Activity Pane of GateKeeper shows many http requests, but none of them are being serviced. Client browsers timeout waiting for a response. Stopping Wingate.exe and restarting fixed the problem this time. This is not a new problem for me. Going all the way back to WG 4.x the www proxy has exhibited this odd behaviour for me. With WG 5.2.3, the www proxy would run correctly for sometimes up to 10 days, then I would need to reboot my win2k server because Wingate.exe would hang while attempting to stop. Versions prior to 5.2.3 would only run for 3 or 4 days before I would have to reboot my win2k server. I was hoping that WG 6 would solve this nagging problem, but is seems that it doesn't and in fact for me it is worse than 5.2.3 as WG 6 lasted only 4 days. I have slow DSL and only 4 client users, so it's not like I'm pushing the limits of WG. Can this issue be fixed for WG 6? I will do what ever you ask to help find and fix this. Thanks, Steve

[Pascal]

Are you running any plugins for WinGate ? At the time when it fails like that - can you check to see what the CPU utilisation is from Task Manager, please ?

Also, to confirm - this is using direct proxy connections or are you using TRs ?

[saubrey]

I have Kapersky AV plug-in running and configured for www proxy and ftp proxy. CPU utilization on my server is minimal. Wingate.exe is not consuming any cpu (or at least very little) when the www proxy hangs. Some clients have configured IE to directly use the WG www proxy, and other clients are using Transparent Redirection. I'm not sure which clients were running today at the time www proxy hung. If you request I can either convert all clients to use TR or I can convert all to use to directly use the proxy

[Pascal]

No, it shouldn't be necessary to convert them all to one connection type at once. Was kavss.exe also running on minimal CPU useage ?

Now, at the time it locks like that - FTP and SOCKS can still go through, correct ? I'm going to try to work out where this happens - because it doesn't sound as if it's connectivity from the client to the WinGate Server OR from the WinGate Server to the outside world.

You can still lookup names, download files, etc. through the FTP proxy, correct ?

Also, you can go offline and online in GateKeeper, with no problems, correct ?

Last question would be - can you see what happens if you create a secondary HTTP Proxy Service at the time when it fails and try to connect one of the clients through that ? (This is a troubleshooting step, to see if it is specifically HTTP Related ot to a specific HTTP Service)

[Pascal]

Is there anything else that might distinguish your setup from a straight, out-of-the-box installation ?

I've setup a test rig (6.0 - 984) with 8 clients connected to it, they're currently all doing HTTP surfing (NAT + TR) through the server (P2 400, 128mb RAM) with KAV and PureSight installed.

However, my results might be a bit skewed if you have authentication / custom rules or anything that might be different from a vanilla install.

[Pascal]

Interesting - for a moment there it looked as if one of our gateways in the office also failed. (Been running 975 since it's release, though) I couldn't find anything wrong the server - it was still responding to pings, NAT traffic was going through, etc.

I rebooted my client though, and it all started working again. On your side - what happens if you reboot the client ?

[saubrey]

>Was kavss.exe also running on minimal CPU useage ?
Yes. There were no processes consuming abnormal cpu. This included Wingate.exe and the two kavss.exe processes

>Now, at the time it locks like that - FTP and SOCKS can still go
>through, correct ?
Yes, FTP, Socks, and NAT all continue to work. Only www server hangs. I also have POP3 proxy and RSTP proxy configured, but I did not notice if they still worked


>I'm going to try to work out where this happens - because it doesn't
>sound as if it's connectivity from the client to the WinGate Server OR
>from the WinGate Server to the outside world.

>You can still lookup names, download files, etc. through the FTP proxy, >correct ?
OK, I admit that this time I did not try FTP to see that it was still working. I am also not certain this time that socks was working. Previous times with 5.2.3, FTP & socks continued to work when www proxy hung as previously I did specifically try them, but I did not try socks and ftp this time. This time I am only certain that NAT continued to work. I suspect that socks continued to work only becuase AOL IM continued to work and most of my computers have AOL IM configured to use socks proxy. However one computer is still configured not to use any proxy for AOL IM...I"m not sure which computer(s) were being used to run AOL IM at the time www proxy hung.

>Also, you can go offline and online in GateKeeper, with no problems, >correct ?
This time, yes. I was able to go offline then online with Gatekeeper. Also I was able to successfully stop, the restart Wingate engine. Howerver, other times with 5.2.3 and pervious versions, only sometimes could I successfully go offline/online with GateKeeper and only sometimes could I successfully stop/starte Wingate engine. Ususally with 5.2.3 after 3 or 4 days of running the www proxy would hang and I could stop/start Wingate engine. Then Wingate would run for another few days, and then www proxy would hang again. Usually with the 2nd hang, I could not stop Wingate engine, I would have to reboot win2k.

>can you see
>what happens if you create a secondary HTTP Proxy Service at the
>time when it fails and try to connect one of the clients through that ?
OK, no problem I will try this. BTW I already have two www proxies configured...both stopped working this time. The 2nd www proxy is new for me...I've only been using it for the past month. The 2nd www proxy is configured to accept connections on my external NIC card...when the 1st proxy hung, I tried to connect to the 2nd proxy via the external interface and was unable to, so I assumed that it hung as well. Previously with 5.2.3, when I only had 1 www proxy configured, that single www proxy would hang after 4 - 10 days of running.

>What else might distinguish your setup
I run IIS 5.0 with www server and FTP server.
I run Argosoft smpt/pop3 email server
I run BlackIce Server Protection
All are run on the same win2k server as Wingate.
I run Win2k w/Active Directory
I don't run Puresight. I don't run WG's VPN. I have turned off WG's: DHCP, DNS, Winsock redirector service, pop3 and smtp server . I've got about 20 items defined in WG's port security firewall either to explicitly deny some ports or explicity open them, or to redirect them. I could send you my WG registry entries if you like

[Pascal]

Just managed to get this to happen. So, if this happens for you I want you to check two things please. I'm not 100% sure if these things are related, but they are abnormalities I noticed on the setup I was using:

1. Check the amount of free disk space on the server. In the test case, there were about 20 odd web sessions trying to download files of between 8 and 15 mb each, but logging,history and quarantine overnight had driven it down to almost no free space.

2. Run "netstat -an" from a command line. This machine had an abnormal amount of listening sockets - all TCP based. This would not be normal for web traffic, but not currently sure if its related to the out-of-disk space scenario or not.

[saubrey]

OK. I'll give an update in 4 - 10 days when it happens again

[saubrey]

3 days 23 hours and then the www proxy hung. I have all the info that you requested, and more. NAT, FTP, socks5, rstp continue to run correctly...I verified all of them. No abnormal CPU utilization by any process on my server, including the WG processes. Hard drives have 800 MB to 2 GB availalbe! I created a new www proxy and it fails just like the first two that I have. netstat -an (see below). As an experiment I turned off TR in my exising www proxy and reconfigured my client browser to not use a proxy and the www proxy then correctly serviced the browser's request. I then turned back on TR, and left the browser not to use a proxy, and www proxy failed to service requests. I then turned off TR and then the www proxy worked correctly again. I then, with TR still off, configured my browser to use a proxy, and then the www proxy failed to service requests. I think the bug is in the www proxy. What next should I do?




D:\Documents and Settings>netstat -an

Active Connections

Proto Local Address Foreign Address State
TCP 0.0.0.0:7 0.0.0.0:0 LISTENING
TCP 0.0.0.0:9 0.0.0.0:0 LISTENING
TCP 0.0.0.0:13 0.0.0.0:0 LISTENING
TCP 0.0.0.0:17 0.0.0.0:0 LISTENING
TCP 0.0.0.0:19 0.0.0.0:0 LISTENING
TCP 0.0.0.0:21 0.0.0.0:0 LISTENING
TCP 0.0.0.0:25 0.0.0.0:0 LISTENING
TCP 0.0.0.0:42 0.0.0.0:0 LISTENING
TCP 0.0.0.0:53 0.0.0.0:0 LISTENING
TCP 0.0.0.0:88 0.0.0.0:0 LISTENING
TCP 0.0.0.0:135 0.0.0.0:0 LISTENING
TCP 0.0.0.0:389 0.0.0.0:0 LISTENING
TCP 0.0.0.0:443 0.0.0.0:0 LISTENING
TCP 0.0.0.0:445 0.0.0.0:0 LISTENING
TCP 0.0.0.0:464 0.0.0.0:0 LISTENING
TCP 0.0.0.0:636 0.0.0.0:0 LISTENING
TCP 0.0.0.0:1032 0.0.0.0:0 LISTENING
TCP 0.0.0.0:1068 0.0.0.0:0 LISTENING
TCP 0.0.0.0:1080 0.0.0.0:0 LISTENING
TCP 0.0.0.0:1081 0.0.0.0:0 LISTENING
TCP 0.0.0.0:1082 0.0.0.0:0 LISTENING
TCP 0.0.0.0:1083 0.0.0.0:0 LISTENING
TCP 0.0.0.0:1229 0.0.0.0:0 LISTENING
TCP 0.0.0.0:1234 0.0.0.0:0 LISTENING
TCP 0.0.0.0:1287 0.0.0.0:0 LISTENING
TCP 0.0.0.0:1325 0.0.0.0:0 LISTENING
TCP 0.0.0.0:1339 0.0.0.0:0 LISTENING
TCP 0.0.0.0:1344 0.0.0.0:0 LISTENING
TCP 0.0.0.0:1356 0.0.0.0:0 LISTENING
TCP 0.0.0.0:1378 0.0.0.0:0 LISTENING
TCP 0.0.0.0:1381 0.0.0.0:0 LISTENING
TCP 0.0.0.0:1385 0.0.0.0:0 LISTENING
TCP 0.0.0.0:1547 0.0.0.0:0 LISTENING
TCP 0.0.0.0:1573 0.0.0.0:0 LISTENING
TCP 0.0.0.0:1580 0.0.0.0:0 LISTENING
TCP 0.0.0.0:1584 0.0.0.0:0 LISTENING
TCP 0.0.0.0:1915 0.0.0.0:0 LISTENING
TCP 0.0.0.0:2058 0.0.0.0:0 LISTENING
TCP 0.0.0.0:2059 0.0.0.0:0 LISTENING
TCP 0.0.0.0:2060 0.0.0.0:0 LISTENING
TCP 0.0.0.0:2061 0.0.0.0:0 LISTENING
TCP 0.0.0.0:2062 0.0.0.0:0 LISTENING
TCP 0.0.0.0:2064 0.0.0.0:0 LISTENING
TCP 0.0.0.0:2077 0.0.0.0:0 LISTENING
TCP 0.0.0.0:2125 0.0.0.0:0 LISTENING
TCP 0.0.0.0:3268 0.0.0.0:0 LISTENING
TCP 0.0.0.0:3269 0.0.0.0:0 LISTENING
TCP 0.0.0.0:3372 0.0.0.0:0 LISTENING
TCP 0.0.0.0:3389 0.0.0.0:0 LISTENING
TCP 0.0.0.0:4211 0.0.0.0:0 LISTENING
TCP 0.0.0.0:4214 0.0.0.0:0 LISTENING
TCP 0.0.0.0:4621 0.0.0.0:0 LISTENING
TCP 0.0.0.0:8080 0.0.0.0:0 LISTENING
TCP 0.0.0.0:8081 0.0.0.0:0 LISTENING
TCP 0.0.0.0:8111 0.0.0.0:0 LISTENING
TCP 66.159.225.165:21 0.0.0.0:0 LISTENING
TCP 66.159.225.165:389 66.159.225.165:4214 ESTABLISHED
TCP 66.159.225.165:1032 66.159.225.165:1344 ESTABLISHED
TCP 66.159.225.165:1032 66.159.225.165:1915 ESTABLISHED
TCP 66.159.225.165:1344 66.159.225.165:1032 ESTABLISHED
TCP 66.159.225.165:1547 66.159.225.165:389 CLOSE_WAIT
TCP 66.159.225.165:1573 207.188.7.202:554 ESTABLISHED
TCP 66.159.225.165:1580 64.12.24.226:5190 ESTABLISHED
TCP 66.159.225.165:1584 64.12.27.177:5190 ESTABLISHED
TCP 66.159.225.165:1915 66.159.225.165:1032 ESTABLISHED
TCP 66.159.225.165:2058 128.121.104.48:80 CLOSE_WAIT
TCP 66.159.225.165:2059 128.121.104.48:80 CLOSE_WAIT
TCP 66.159.225.165:2060 128.121.104.48:80 CLOSE_WAIT
TCP 66.159.225.165:2061 128.121.104.48:80 CLOSE_WAIT
TCP 66.159.225.165:2062 128.121.104.48:80 CLOSE_WAIT
TCP 66.159.225.165:2064 128.121.104.48:80 CLOSE_WAIT
TCP 66.159.225.165:2077 205.188.139.152:80 CLOSE_WAIT
TCP 66.159.225.165:4214 66.159.225.165:389 ESTABLISHED
TCP 66.159.225.165:4621 66.159.225.165:389 CLOSE_WAIT
TCP 66.159.225.165:8091 0.0.0.0:0 LISTENING
TCP 127.0.0.1:21 0.0.0.0:0 LISTENING
TCP 127.0.0.1:80 0.0.0.0:0 LISTENING
TCP 127.0.0.1:110 0.0.0.0:0 LISTENING
TCP 127.0.0.1:389 127.0.0.1:1080 ESTABLISHED
TCP 127.0.0.1:389 127.0.0.1:1081 ESTABLISHED
TCP 127.0.0.1:389 127.0.0.1:1083 ESTABLISHED
TCP 127.0.0.1:389 127.0.0.1:4211 ESTABLISHED
TCP 127.0.0.1:554 0.0.0.0:0 LISTENING
TCP 127.0.0.1:808 0.0.0.0:0 LISTENING
TCP 127.0.0.1:808 127.0.0.1:2125 ESTABLISHED
TCP 127.0.0.1:1080 0.0.0.0:0 LISTENING
TCP 127.0.0.1:1080 127.0.0.1:389 ESTABLISHED
TCP 127.0.0.1:1081 127.0.0.1:389 ESTABLISHED
TCP 127.0.0.1:1083 127.0.0.1:389 ESTABLISHED
TCP 127.0.0.1:1234 127.0.0.1:389 CLOSE_WAIT
TCP 127.0.0.1:2125 127.0.0.1:808 ESTABLISHED
TCP 127.0.0.1:4211 127.0.0.1:389 ESTABLISHED
TCP 127.0.0.1:8010 0.0.0.0:0 LISTENING
TCP 127.0.0.1:8091 0.0.0.0:0 LISTENING
TCP 192.168.0.2:21 0.0.0.0:0 LISTENING
TCP 192.168.0.2:80 0.0.0.0:0 LISTENING
TCP 192.168.0.2:80 192.168.0.126:2138 CLOSE_WAIT
TCP 192.168.0.2:80 192.168.0.126:2142 ESTABLISHED
TCP 192.168.0.2:80 192.168.0.126:2143 ESTABLISHED
TCP 192.168.0.2:110 0.0.0.0:0 LISTENING
TCP 192.168.0.2:139 0.0.0.0:0 LISTENING
TCP 192.168.0.2:554 0.0.0.0:0 LISTENING
TCP 192.168.0.2:554 192.168.0.126:1914 ESTABLISHED
TCP 192.168.0.2:1080 0.0.0.0:0 LISTENING
TCP 192.168.0.2:1080 192.168.0.126:1922 ESTABLISHED
TCP 192.168.0.2:1080 192.168.0.126:1928 ESTABLISHED
TCP 192.168.0.2:3389 192.168.0.126:2149 ESTABLISHED
TCP 192.168.0.2:8010 0.0.0.0:0 LISTENING
TCP 192.168.0.2:8091 0.0.0.0:0 LISTENING
UDP 0.0.0.0:7 *:*
UDP 0.0.0.0:9 *:*
UDP 0.0.0.0:13 *:*
UDP 0.0.0.0:17 *:*
UDP 0.0.0.0:19 *:*
UDP 0.0.0.0:42 *:*
UDP 0.0.0.0:68 *:*
UDP 0.0.0.0:161 *:*
UDP 0.0.0.0:445 *:*
UDP 0.0.0.0:1065 *:*
UDP 0.0.0.0:1075 *:*
UDP 0.0.0.0:1077 *:*
UDP 0.0.0.0:1079 *:*
UDP 0.0.0.0:1340 *:*
UDP 0.0.0.0:1347 *:*
UDP 0.0.0.0:1348 *:*
UDP 0.0.0.0:1384 *:*
UDP 0.0.0.0:1390 *:*
UDP 0.0.0.0:1391 *:*
UDP 0.0.0.0:1392 *:*
UDP 0.0.0.0:1393 *:*
UDP 0.0.0.0:1488 *:*
UDP 0.0.0.0:1955 *:*
UDP 0.0.0.0:2093 *:*
UDP 0.0.0.0:2975 *:*
UDP 0.0.0.0:3080 *:*
UDP 0.0.0.0:3456 *:*
UDP 0.0.0.0:38037 *:*
UDP 0.0.0.0:38293 *:*
UDP 66.159.225.165:53 *:*
UDP 66.159.225.165:67 *:*
UDP 66.159.225.165:68 *:*
UDP 66.159.225.165:88 *:*
UDP 66.159.225.165:123 *:*
UDP 66.159.225.165:389 *:*
UDP 66.159.225.165:464 *:*
UDP 66.159.225.165:500 *:*
UDP 66.159.225.165:2535 *:*
UDP 66.159.225.165:4500 *:*
UDP 127.0.0.1:53 *:*
UDP 127.0.0.1:368 *:*
UDP 127.0.0.1:1346 *:*
UDP 192.168.0.2:53 *:*
UDP 192.168.0.2:67 *:*
UDP 192.168.0.2:68 *:*
UDP 192.168.0.2:88 *:*
UDP 192.168.0.2:123 *:*
UDP 192.168.0.2:137 *:*
UDP 192.168.0.2:138 *:*
UDP 192.168.0.2:368 *:*
UDP 192.168.0.2:389 *:*
UDP 192.168.0.2:464 *:*
UDP 192.168.0.2:500 *:*
UDP 192.168.0.2:2535 *:*
UDP 192.168.0.2:4500 *:*

D:\Documents and Settings>

[Pascal]

If you are using NAT, without TR enabled, then it won't be using plugins. I suspect that you are getting stuck when Kaspersky AntiVirus updates. There are automatic events set to update at certain times during the day (Put there by the installer - you can double check them in Scheduler).

Under moderate load (9 machines running about 3 - 5 web browsers 24/7 each) I've seen it block all WWW activity about 1 in 5 updates. I'm having it run updates once per hour - and when I walk past the server I run another one just for the heck of it.

If it seems to coincide with automatic av database updates on your end too - try turning them off (OR reschedule them to a time when everybody in the office / at home is asleep)

This will 'cure' it while we work on a fix. We are busy testing a new SDK from Kaspersky Labs, which seems to have resolved this issue on our side - but it will definately require extensive retesting first.

[saubrey]

I am currently auto updating Kaspersky every 90 minutes...I'll reduce it to once per day at night, and see what happens. Thanks.

[saubrey]

After changing AV to update only once a day during the night, the WWW proxy has not stopped servicing requests. I'm now up to straight 10 days of up time with Wingate and have not had any problems with the WWW proxy. I'll let Wingate and my Win2k server run for as long as they can to help determine if the AV updating was the root cause of the WWW proxy not servicing requests problem. Thanks for your help

[Pascal]

We've spoken to Kaspersky, and they've released a new version of their SDK and libraries which fixes the update problem. It's not ready for release, but will be in the near future, once development and QA on it are done.