Wingate frequent hanging

[pecos]

I've been having problems with Wingate freezing up since version 4--I kept using 4.5.1 for a long time because with that version all that was needed to restore functionality was to hang up and redial the Internet connection (with later versions, rebooting the server machine was required).

I finally tried version 5.2.0 and it ran (nearly) flawlessly for a couple of months, but then it started freezing up often. I upgraded to 5.2.2, but the problem became even worse--even after doing a complete uninstall and restoring 5.2.0. I tried various actions besides rebooting to try to get the engine to start responding, such as stopping and restarting the WWW proxy server, turning logging on or off, and stopping and restarting the Wingate engine. It seemed that some of these actions would get it to start responding again, but I discovered that eventually Wingate would just start working again on its own. The problem has mainly been noted on the server machine, but sometimes the client machine (there is only one) will be able to receive web pages when the server machine is unable to. The frezeups do not seem to affect email at all.

I tried updating the NIC driver (3Com 3C905C) on the server machine, but it still freezes up. Does anyone have any other suggestions? I generally keep logging, caching, and history all disabled; deleting the history files does not seem to have any effect.

--Tom

[Pascal]

When the engine was in a "frozen" state, could you still log-in with GateKeeper and even shut it down ?

Was there a specific error message you got from the client ?

The main thing I'm trying to determine is if it is a lock-up inside the software (Dead-lock) or if it is a problem with internet connectivity.

If it is deadlock related, we can enable a few additional logging fields that will tell us with reasonable certainty exactly where the problem lies.

[pecos]

When the engine was in a "frozen" state, could you still log-in with GateKeeper and even shut it down ?

Yes. (In fact, just as I finished writing this post, Wingate froze up. I opened the WWW proxy server properties to turn on logging, and it began working again.)

Was there a specific error message you got from the client ?

In Opera on the server machine, I get "Socket Error. Connection to Remote Host timed out." On the client machine (using IE), "The requested page could not be displayed."

The main thing I'm trying to determine is if it is a lock-up inside the software (Dead-lock) or if it is a problem with internet connectivity.

I'm pretty sure it's not an Internet connectivity issue. As I said, sometimes the client machine can connect (using NAT) when the server machine cannot. If I uncheck the http proxy in Opera I can sometimes (but not always) connect that way when Wingate is not responding.

[Pascal]

I'm pretty sure it's not an Internet connectivity issue. As I said, sometimes the client machine can connect (using NAT) when the server machine cannot. If I uncheck the http proxy in Opera I can sometimes (but not always) connect that way when Wingate is not responding.

This test with Opera, is it on the client or on the server ? 99% of the time you should be able to get out from the server if your browser is not set to proxy.

Hopefully it reaches that state again soon, because if you can grab (a) a screenshot in TaskManager showing the resource utilisation for WinGate (Or just write it down) and (b) a capture of netstat -an to show us how many sockets are in-use, that would be a usefull thing.

[pecos]

This test with Opera, is it on the client or on the server ? 99% of the time you should be able to get out from the server if your browser is not set to proxy.

On the server. It used to work; I'm not sure why it stopped.

Hopefully it reaches that state again soon, because if you can grab (a) a screenshot in TaskManager showing the resource utilisation for WinGate (Or just write it down) and (b) a capture of netstat -an to show us how many sockets are in-use, that would be a usefull thing.

I've never seen Wingate using much more than 5% of the CPU or 5,600K of memory in Task Manager, even when locked up. I'm not sure how to get a capture of netstat, sorry. But I just ran it and see quite a few ports listed as "Listening" and two as "Established."

[Pascal]

I've never seen Wingate using much more than 5% of the CPU or 5,600K of memory in Task Manager, even when locked up. I'm not sure how to get a capture of netstat, sorry. But I just ran it and see quite a few ports listed as "Listening" and two as "Established."

netstat -an > capture.txt

will save the interesting information to a file called "capture.txt"

It'd only be a concern if there were 100s or 1000s of connections listed with CLOSE_WAIT, TIME_WAIT or similar.

[pecos]

netstat -an > capture.txt

Oh, yeah--DOS commands. I used to know a couple of those a long time ago. :-)

It'd only be a concern if there were 100s or 1000s of connections listed with CLOSE_WAIT, TIME_WAIT or similar.

Well, only a couple of TIME_WAITs. Task Manager showed CPU usage of 0%-2% and memory usage of 4,300K.

Active Connections

Proto Local Address Foreign Address State
TCP 0.0.0.0:135 0.0.0.0:0 LISTENING
TCP 0.0.0.0:445 0.0.0.0:0 LISTENING
TCP 0.0.0.0:1025 0.0.0.0:0 LISTENING
TCP 0.0.0.0:1028 0.0.0.0:0 LISTENING
TCP 0.0.0.0:3037 0.0.0.0:0 LISTENING
TCP 0.0.0.0:3178 0.0.0.0:0 LISTENING
TCP 0.0.0.0:5000 0.0.0.0:0 LISTENING
TCP 127.0.0.1:21 0.0.0.0:0 LISTENING
TCP 127.0.0.1:23 0.0.0.0:0 LISTENING
TCP 127.0.0.1:80 0.0.0.0:0 LISTENING
TCP 127.0.0.1:110 0.0.0.0:0 LISTENING
TCP 127.0.0.1:554 0.0.0.0:0 LISTENING
TCP 127.0.0.1:808 0.0.0.0:0 LISTENING
TCP 127.0.0.1:808 127.0.0.1:3178 ESTABLISHED
TCP 127.0.0.1:1027 0.0.0.0:0 LISTENING
TCP 127.0.0.1:1080 0.0.0.0:0 LISTENING
TCP 127.0.0.1:2080 0.0.0.0:0 LISTENING
TCP 127.0.0.1:3178 127.0.0.1:808 ESTABLISHED
TCP 127.0.0.1:7000 0.0.0.0:0 LISTENING
TCP 127.0.0.1:8010 0.0.0.0:0 LISTENING
TCP 192.168.0.1:21 0.0.0.0:0 LISTENING
TCP 192.168.0.1:23 0.0.0.0:0 LISTENING
TCP 192.168.0.1:80 0.0.0.0:0 LISTENING
TCP 192.168.0.1:80 192.168.0.1:3187 TIME_WAIT
TCP 192.168.0.1:110 0.0.0.0:0 LISTENING
TCP 192.168.0.1:139 0.0.0.0:0 LISTENING
TCP 192.168.0.1:554 0.0.0.0:0 LISTENING
TCP 192.168.0.1:1080 0.0.0.0:0 LISTENING
TCP 192.168.0.1:2080 0.0.0.0:0 LISTENING
TCP 192.168.0.1:3191 192.168.0.2:139 TIME_WAIT
TCP 192.168.0.1:7000 0.0.0.0:0 LISTENING
TCP 192.168.0.1:8010 0.0.0.0:0 LISTENING
UDP 0.0.0.0:445 *:*
UDP 0.0.0.0:500 *:*
UDP 0.0.0.0:1064 *:*
UDP 0.0.0.0:1065 *:*
UDP 0.0.0.0:1066 *:*
UDP 0.0.0.0:1067 *:*
UDP 0.0.0.0:1706 *:*
UDP 63.168.39.163:123 *:*
UDP 63.168.39.163:1900 *:*
UDP 127.0.0.1:123 *:*
UDP 127.0.0.1:368 *:*
UDP 127.0.0.1:1068 *:*
UDP 127.0.0.1:1900 *:*
UDP 127.0.0.1:8000 *:*
UDP 192.168.0.1:53 *:*
UDP 192.168.0.1:67 *:*
UDP 192.168.0.1:123 *:*
UDP 192.168.0.1:137 *:*
UDP 192.168.0.1:138 *:*
UDP 192.168.0.1:368 *:*
UDP 192.168.0.1:1068 *:*
UDP 192.168.0.1:1900 *:*
UDP 192.168.0.1:8000 *:*

[norita]

Every time I logon to Wingate Gatekeeper its hang. In the history it did not record the last couple weeks records and it show only the one on the 4 february 2004.

I have tried to uninstall and reinstall, it is still the same.

The wingate engine are still working properly.

Can anyone help me with this.

Norita

[pecos]

If I uncheck the http proxy in Opera I can sometimes (but not always) connect that way when Wingate is not responding.

99% of the time you should be able to get out from the server if your browser is not set to proxy.

Yesterday it worked, but today it didn't. I would get "Network error: Could not connect to server." But then when the proxy began responding I could connect again--even without the proxy enabled in the browser.

Is there anything else I should be trying? Should I send log excerpts? Could this be a NIC problem?

--Tom

[Pascal]

This problem crops up for a few users. Have a look at the posts towards the bottom http://forums.qbik.com/viewtopic.php?t=135.

riaanjvr has the same problem and seems to have found a work-around. You could try that in the interim, while we nail it down in the code.

[pecos]

riaanjvr has the same problem and seems to have found a work-around. You could try that in the interim, while we nail it down in the code.

Thanks, I hadn't seen that post before. I already had caching disabled, but I applied his other suggestions about the file size and the blank filter. I'll be interested to see if these changes make any difference.

[Pascal]

Thanks, I hadn't seen that post before. I already had caching disabled, but I applied his other suggestions about the file size and the blank filter. I'll be interested to see if these changes make any difference.

Excellent. If it does solve it, please post back, I'd be interested in hearing the results.

Would you by any chance be interested in running a debug kit to help narrow this problem down for us, please ?

The process involved in a debug kit is fairly simple. For a certain timeframe (Usually until the problem occurs) you run a copy of WinGate and GateKeeper that performs more extensive logging to file. Once the error occurs, the log should contain sufficient information to tell us where it is, at which stage you can revert to your previous executables.

[pecos]

I'll let you know how it goes.

I'd be happy to run the debug kit. I ran one some time ago with version 4 trying to solve a similar problem.

--Tom

[Pascal]

I'll let you know how it goes.

I'd be happy to run the debug kit. I ran one some time ago with version 4 trying to solve a similar problem.

--Tom

Thanks Tom, the debug kit is approximately 2MB zipped, can I email it to you directly ? (Address in profile ?)

[pecos]

That would be fine, Pascal.

--Tom

[pecos]

I don't know if 5.2.3 addressed this problem, but I installed this update and it does not perform any better than 5.2.2. In fact, the performance of Wingate seems to have deteriorated in recent weeks; it now sometimes requires a reboot of the server machine to recover from freezeups.

--Tom

[adrien]

Hi

Given the fact that it worked well for a couple of months then started to deteriorate, that would indicate to me that something outside of the WinGate executables is changing which is affecting the performance.

The most likely culprit for this is the history database file.

If you

1. Stop the engine
2. delete the files history.dbf and history.cdx from the WinGate folder
3. restart the engine

and see if that improves things.

Adrien

[adrien]

PS. Once the history file gets near the auto purge level, then you can end up with frequent history purges, which can slow things down a lot depending on the size of the file.

We find in the end, the default history database file size is way too big, and recommend normally that this be decreased to about 5M from the 50M default setting, since GateKeeper only displays 2000 entries anyway.

Adrien

[pecos]

Thanks adrien. The only problem I have with following your suggestion is that I have disabled history recording.

--Tom

[adrien]

OK, so are the files small? Like 3k each or so?

Adrien

[pecos]

Yes, History.dbf is 1K and History.cdx is 3K.

I deleted the files and enabled history recording with the reduced file size you recommended, and Wingate froze up almost immediately afterward. So I reverted to the disabled history setting.

--Tom

[pecos]

I've been using Wingate since version 3.0, but I'm getting really frustrated with this problem, which has existed for a very long time now. I've been dealing with it, but it appears to be only getting worse. I'm just about ready to start using Windows Internet Connection Sharing.

--Tom

[adrien]

Hi

Understand the frustration. I am very suspicious that you say that the problem has gotten worse though.

This makes me wonder whether there is something new happening on your network to create problems like this.

Have you recently looked for viruses on your network? Things like the MS blaster virus can wreak a bit of havoc if left unchecked.

One other thing I can suggest is to turn on debug deadlock detection, so that if there is a deadlock in the WinGate code making it freeze, then we will get an indication of where in the code is the problem. To turn this on, go to regedit, and add a value under

HKEY_LOCAL_MACHINE\Software\Qbik Software\WinGate\Settings

create a value called "UseDeadlockDetection" type REG_DWORD and set it to 1.
Same with another value called "CheckLockPrecedence".

If you get a file called LockAnalysisDumpFileEng.txt created in the WinGate directory, email it to me directly at adrien@qbik.com

One other thing to check. If WinGate is not actually locking up, but instead is ceasing to respond for a while, that normally indicates a DNS issue in the WinGate DNS resolver. If WinGate (or the OS winGate is running on) knows about more than one DNS server, then depending on whether all the known DNS servers are responsive or not, there can be problems. In particular if you are running WinGate's DNS server

1. Make sure that the OS DNS settings do not include the addresses of any of your network adapters
2. Make sure that the DNS servers specified are all known good DNS servers.

Regards

Adrien

[pecos]

Have you recently looked for viruses on your network? Things like the MS blaster virus can wreak a bit of havoc if left unchecked.

I keep my virus definitions current and scan often. However, the return of the problem (after a period of good functionality) did seem to coincide with the installation of Norton Systemworks Pro 2004. I'm reluctant to uninstall it for testing purposes, though, because I do not have the Wingate antivirus plugin installed (can it be installed on a trial basis)?

If you get a file called LockAnalysisDumpFileEng.txt created in the WinGate directory, email it to me directly

I haven't enabled lockup analysis recently, but I did run a debug kit for Pascal a week or two ago.

One other thing to check. If WinGate is not actually locking up, but instead is ceasing to respond for a while, that normally indicates a DNS issue in the WinGate DNS resolver.

That sounds like what is happening, because the behavior mainly affects the server machine and generally resolves itself after a period of time.

2. Make sure that the DNS servers specified are all known good DNS servers.

Is there some way to get a list of known good DNS servers that can be used? I have tried entering my ISP's server, Qbik's, and Microsoft's (and none) in the DNS Resolver properties, but none of these settings seem to affect the behavior.

[pecos]

Slight update:

I discovered that there was a problem with my installation of Norton Systemworks, so I uninstalled and reinstalled it. However, this did not bring an end to the Wingate behavior.

I have also noticed a few other details about how Wingate behaves when it is nonresponsive:

1. Socket errors appear in the browser 20-22 seconds after initiation of the request, although Wingate service timeouts are set to 60 seconds (except for Winsock Redirector Service, which is set at 20 seconds; stopping this service seemed to revive Wingate once, but this was not repeatable).

2. Changing the timeout settings for the various Wingate services does not affect the timing of the socket errors.

3. In the Opera status bar, "Receiving data from server" is briefly displayed just before socket errors appear.

4. Wingate freezeups do not appear to affect the Opera email client, but they do affect Outlook Express.

[andrewclark]

[quote="pecos"]Slight update:

>I discovered that there was a problem with my installation of Norton Systemworks, so I uninstalled and reinstalled it. However, this did not bring an end to the Wingate behavior.


If I could just add my pennyworth here...

I too am having problems with Wingate (5.2.2) on my new XP Pro server with it hanging the whole system usually, but not always, when left running overnight.

I also have Norton installed (Internet Security 2003), and last night I noticed the machine had failed again at 0154 hours, just 5 mins before the AV scan is cheduled to kick in.

I found this in the System logs - the last entry before this mornings reboot was at that time.

However the logs don't make much sense to me (they used to be in plain text but now look like this):

02/28/04 01:54:01 Uik7NVIpPeQS5ACHAQAAAAYADWEVWmoACkJuUFQAAEUAADDvg0AAfwbu2FIpOzVSKT3kEuQAh7BNk+sAAAAAcAL68BMfAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
02/28/04 01:54:02 Uik7NVIpPeQS4QCHAQAAAAYADWEVWmoACkJuUFQAAEUAADDviEAAfwbu01IpOzVSKT3kEuEAh7BEqYMAAAAAcAL68P2SAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
02/28/04 01:54:04 Uik7NVIpPeQS5ACHAQAAAAYADWEVWmoACkJuUFQAAEUAADDvjkAAfwbuzVIpOzVSKT3kEuQAh7BNk+sAAAAAcAL68BMfAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
02/28/04 01:54:08 Uik7NVIpPeQS4QCHAQAAAAYADWEVWmoACkJuUFQAAEUAADDvoUAAfwbuulIpOzVSKT3kEuEAh7BEqYMAAAAAcAL68P2SAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
02/28/04 01:54:10 Uik7NVIpPeQS5ACHAQAAAAYADWEVWmoACkJuUFQAAEUAADDvpkAAfwbutVIpOzVSKT3kEuQAh7BNk+sAAAAAcAL68BMfAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=

Can anyone throw any light on
1) What might be hanging the entire server
2) How to make any sense of the logs when it fails!

Regards
Andrew

[pecos]

Andrew,

There is a thread on indecipherable system logs here. (It's base 64 code.)

Adrien, I enabled lockup analysis yesterday; I'll send you the log it has generated so far.

--Tom

[andrewclark]

>Andrew,

>There is a thread on indecipherable system logs here. (It's base 64 code.)

Tom

Thanks for that. It appears the logs are no longer meaninful to us ordinary mortals!

I also found something else interesting in that thread - Symantec AV should be set to Exclude from scanning the Wingate Logs as they change every few seconds and have then to be re-scanned. I'm going to try that and see if it stops Wingate hanging the server.

[pecos]

Andrew,

I don't quite understand the point of putting such information in a log either.

Thanks for pointing out the Norton Antivirus part. I must have missed that when I looked before. Since I generally keep logging tuned off I'm not sure how much it'll help me, but I'll give it a try.

[andrewclark]

Andrew,

I don't quite understand the point of putting such information in a log either.

Thanks for pointing out the Norton Antivirus part. I must have missed that when I looked before. Since I generally keep logging tuned off I'm not sure how much it'll help me, but I'll give it a try.

Update:

Sadly, that didn't work either. Wingate has just hung the whole O/S again. I had just reinstalled 5.2.2 in case the previous reboots had damaged it. The annoying thing is that there is no apparent trace of what is causing it to hang - even in the XP Event Log. The last entry, though, looks like Norton AV:
"The NAVENG service was successfully sent a start control."
I wonder if there are any known conflicts with Norton? I previously used Wingate with McAfee, on my old server.

Rgds
Andrew