NAT & RAM?

[Bill.Bowen]

Our Wingate 5 Enterprise Server needs to be restarted daily. Sometimes more! Yes, it is running the latest software. NAT traffic stopping is the first indicator of the problem. Before restarting, I try to examine the Wingate Engine in Task Scheduler. On our server, we have 1Gb of RAM. When Wingate stops responding, its memory usage is always at leat 600,000K. I have seen it as high as 715,000K. When I restart the engine, it's initial size is about 15,000k but it continually grows in 8-12K increments. What's the deal? Is there a memory leak or something? Can it be fixed???

To temporarily combat this source of aggravation, I have scheduled a task to restart the engine daily at 0500. Obviously, this is a bandaid.

[Bill.Bowen]

Hmmm... Restarting doesn't seem to fix the problem either. My Wingate Server still forwards all proxies (IMAP, SMTP, WWW, etc) but authentication stop working. Users cannot connect with GateKeeper or WGIC! The coexisting IIS webserver can still authenticate though. It seems a reboot is the only fix. The memory usage is still astronomical, too. The server is a Dell SC600 P4-2Gb Server with 1Gb RAM running Windows 2000 Server (with latest updates), Symantec AntiVirus 8.1 client, and Wingate 5.2.2 (unlimited users) with two plugins (25-user KAV & Puresight). I have disabled the Symantec software and still have this problem and am running out of ideas. Anyone else have this problem? Any suggestions Qbik? Any help would be greatly appreciated!

[Pascal]

AntiVirus 8.1 client, and Wingate 5.2.2 (unlimited users) with two plugins (25-user KAV & Puresight). I have disabled the Symantec software and still have this problem and am running out of ideas. Anyone else have this problem? Any suggestions Qbik? Any help would be greatly appreciated!

There are a few areas that could affect this. We're not aware of any memory leaks at the moment, but we could try a few different things to try and narrow it down.

The places to check are:

(a) Size of www-cache: Index of cache could affect memory usage
(b) Size of dns-cache: you could purge this, see how much memory it releases
(c) History and Logging: Try turning off those items you do not need

As you're running both plugins, ensure you have the latest version of both of them. We did have a problem with Kaspersky AntiVirus and excessive memory usage quite recently - about a month ago - when it is applied to WWW scanning. We've since fixed that. You can try disabling the plugins individually on the services you use most often to see if that is the case.

[Bill.Bowen]

Ok, thanks. I've knocked the cache size down to 100M from 200M. Checked my KAV plugin, too. Says it's 1.2.1 so it will need updated to 1.2.2. I thought 1.2.2 was what I installed but will reaccomplish anyway. Shouldn't the AutoUpdate for the plugin keep it up-to-date or is that simply for signature files?

[Pascal]

Ok, thanks. I've knocked the cache size down to 100M from 200M. Checked my KAV plugin, too. Says it's 1.2.1 so it will need updated to 1.2.2. I thought 1.2.2 was what I installed but will reaccomplish anyway. Shouldn't the AutoUpdate for the plugin keep it up-to-date or is that simply for signature files?

Only for signature files at the moment, sorry.

[Bill.Bowen]

Updated to 1.2.2. Appears that the memory consumption problem has been licked but I'm still routinely crashing. Now 2-3 times per day! The Wingate Server just stops responding. No messages in the logs (Win2K or Wingate). Logging is at a minimum. Caching disabled. Win2k box rebuilt. No unnecessary services running.

I'm left with having to remove the KAV & PureSight plugins to see if any sort of stability is attainable. With over a 100 users, I've got to find a fix. C'mon Qbik! I can't be the only Enterprise user with these problems.

[Pascal]

I'm left with having to remove the KAV & PureSight plugins to see if any sort of stability is attainable. With over a 100 users, I've got to find a fix. C'mon Qbik! I can't be the only Enterprise user with these problems.

Is "crashing" a blue-screen of death or can you still use the machine at all ? If it appears that WinGate has simply 'hung' (I.e. you cannot login with GK, stop the engine, etc.) then it is most likely a deadlock.

If that is the case, there are some registry keys we can turn on that will tell us where the deadlock is. It does slow the operation of WinGate down a bit (About 2% -> 5%), but it points to the problem rather quickly.

If it does seem like a dead lock, then you can add the following:

(Remember, all registry warnings apply - so make sure you have a backup / system restore point)

HKEY_LOCAL_MACHINE\Software\Qbik Software\WinGate\Settings

Name: UseDeadlockDetection
Type: REG_DWORD
Value: 1

Name: CheckLockPrecedence
Type: REG_DWORD
Value: 1

Name: LogUnnamedObjects
Type: REG_DWORD
Value: 0 (For now)

Name: LogAgeWarnings
Type: REG_DWORD
Value: 0 (For now)

This will generate a file, LockAnalyserDumpFileEng.txt, in your WinGate folder. That's the one that's of interest to us.

[Bill.Bowen]

Thanks immensely! Have put in the registry entries and restarted. Don't see the "LockAnalyserDumpFileEng.txt" file in my Wingate directory. Is it created only upon a deadlock conflict? If so, when I have the next problem, should I send the file to you? What do I do with it?

No, I'm not pulling a BSOD. Wingate simply stops responding to client requests. I can stop and start the Wingate service, which is what I have to do to fix it! I have removed the KAV & PureSight. Not only have disabled caching but deleted all files in the cache directory. Haven't had any problems yet but the day isn't over yet...

Again. Thanks for the help.

[Pascal]

Wingate directory. Is it created only upon a deadlock conflict? If so, when I have the next problem, should I send the file to you? What do I do with it?

Should only happen on conflict. And yes, send to me, please.

which is what I have to do to fix it! I have removed the KAV & PureSight. Not only have disabled caching but deleted all files

If you run without any problems, would you mind enabling them one at a time to see which one is causing the problem ? That would help me immensely.

Thanks,

[Bill.Bowen]

Bravo! Haven't had to restart my Wingate since yesterday morning. Its ran longer than ever before. Looks like removing those plug-ins helped! I'm going monitor the server for another few days before making any major changes. Then I'll re-enable caching since its loss is noticeable to my surfing users, then followup with each plugin. I'll keep you posted. Thanks.

[Bill.Bowen]

Darn! My PC Tech had to restart Wingate around 0700 this morning. It simply stopped authenticating users via WGIC and Gatekeeper. No errors indicated in the logs (Wingate or W2K). I compute about 40 hours uptime. Best results so far (6-8 hours) but I need it more reliable than 1-2 days! It didn't generate that "LockAnalyserDumpFileEng.txt" file either. I verified those registry settings again, too. They're correct. Must not be a deadlock issue. Now what??? I've trimmed Wingate down to the bare essentials for our operation...

[Pascal]

that "LockAnalyserDumpFileEng.txt" file either. I verified those registry settings again, too. They're correct. Must not be a deadlock issue. Now what??? I've trimmed Wingate down to the bare essentials for our operation...

Do you leave GateKeeper logged in from any location ? Or is WinGate running silently in the background ?

[Bill.Bowen]

If you mean Gatekeeper left running on the Wingate server, NO. Only Wingate, Symantec AV 8.1, and Microsoft IIS 5.0 are running as services. A few users (2-3) may leave their Gatekeeper's running over night. I do frequently so I can review the System & Firewall Messages first thing in the morning. RCS is set to timeout after 900 seconds but it never seems to disconnect.

I've also noticed a problem with IIS. IIS is bound to the external adapter, ports 21 & 80; FTP & Web proxies are bound to the internal adapter, ports 21 & 80. I have holes in the firewall at ports 21 & 80 allowing external connections. If I stop, then restart the webserver, it complains port 80 is used. I need to stop Wingate first before the webserver will successfully restart. Why is that?

[Pascal]

If you mean Gatekeeper left running on the Wingate server, NO. Only Wingate, Symantec AV 8.1, and Microsoft IIS 5.0 are running as services. A few users (2-3) may leave their Gatekeeper's running over night. I do frequently so I can review the System & Firewall Messages first thing in the morning. RCS is set to timeout after 900 seconds but it never seems to disconnect.

Hmmm. We saw an increase in memory usage yesterday when we had GateKeeper open and were hammering the Server with traffic. Can you try keeping all GateKeeper sessions logged off and see if that helps ?

[Bill.Bowen]

Will do. I monitored Wingate's memory usage yesterday during normal business. It hung around 10,500K in Task Manager. That with 2-4 Gatekeepers connected, approximately 15-20 WGIC clients & 4-6 NAT users hitting the server at any given time. My PC Tech didn't examine the task size this morning but will so in the future.

I'm pondering reactivating the scheduled task that restarts Wingate at 0500 daily. Your thoughts?

Also, could the firewall be causing the problem with IIS?

[Pascal]

I'm pondering reactivating the scheduled task that restarts Wingate at 0500 daily. Your thoughts?
Also, could the firewall be causing the problem with IIS?

Reactivating that task might be a good idea, at least until we have a firm handle and fix on this issue.

With regards to the firewall, I think what is happening there is the port-security action needs to time-out. (I assume you've opened a hole for IIS ?) Your config there looks 100%. If you stop IIS and wait approximately 1 -> 3 minutes before restarting it, that should be sufficient time for it to time-out and WG to forget that any traffic had come in. Of course, that depends on how busy IIS is - if you had traffic within a few seconds (10 -> 30) of stopping IIS, then I'd expect this to be the case.

[Bill.Bowen]

Thanks. Will reactivate that task tonight.

Makes sense. Yes, I have holes in the Wingate firewall at ports 21 & 80 allowing packet with standard timeout values. What is the standard value?

[Pascal]

What is the standard value?

From what I can see in the source code, it should be 4 minutes. I'll confirm with Gene when he gets here though, because he's the driver guru.

[Bill.Bowen]

Sorry for the late response. Been enjoying some much needed time off from the circus...

With the scheduled daily 0500 restart, some stability has been achieved but I still suspect some interaction problem with IIS 5. Saturday afternoon, one of our SW Engineers called complaining Internet access was offline. Had them reset the server (reboot). Wingate performed flawlessly since but I noticed a problem this morning (Tuesday). IIS did not automatically start the FTP & Web servers on the external NIC, ports 21 and 80 respectively. Our website had been down all weekend! When I tried start them via the Internet Service Manager, it wouldn't because the ports were already in use. Being the server was restarted Saturday, the ports should have had plenty of time to timeout. When I stopped the FTP & WWW Proxies, they started. What gives? Is the firewall causing this and how do I stop it? I need IIS to always start. It's some coincidence that the server has ran longer than ever before when IIS 5.0 was not running, don't ya think?

And I have yet to produce any deadlock file! The system logs (Wingate & 2K) indicate nothing! Is there a debugging load I could run or something?

[Bill.Bowen]

Arrgh! Wingate bit the dust again. And I think IIS is the culprit. After getting IIS (Web & FTP) restarted this morning, Wingate died around 1800. Worked fine all weekend (Saturday afternoon thru Tuesday morning) with IIS off; turn IIS on and Wingate eventually stops responding! Didn't even last 12 hours! C'mon, it's got to be more than a mere coincidence. I have experimented with every concievable WWW & FTP Proxy configuration in Wingate (always binding to the internal NIC only, of course) and still get an "Address is already in use" error when attempting to start IIS Web or FTP server with Wingate running. The only way to get IIS to start with Wingate running is to unbind the applicable proxy from all adapters. Tried disabling the firewall, no good. Disabled NAT, no good. Port redirection via ENS, no good. Once again, I'm running out of ideas. I need IIS and Wingate to peacefully coexist on this server! I've got more than $800 worth of plugins (25-user PureSight & KAV with 2-year subscriptions, 12-user VPN) I can't utilize in our enterprise because of the server's instablility. Something has to be done. C'mon Qbik, any help would be greatly appreciated.

[Bill.Bowen]

UPDATE: It's been awhile since last post but nothing has changed configuration-wise with our Wingate. No plugins installed. Very minimal logging. Moved backbone servers traffic (SMTP, IMAP4, POP3, DNS) to a router. Am still plagued with random authentication (WGIC) failures and conflicts with IIS at startup.

Anyway, growing used to the frequent Wingate failures, I failed to notice the debug files that my server recently generated. Maybe these will help diagnose our Wingate dilemma. I've sent them to pascalv@qbik.com. Thanks.

[Pascal]

Anyway, growing used to the frequent Wingate failures, I failed to notice the debug files that my server recently generated. Maybe these will help diagnose our Wingate dilemma.

Unfortunately, not. Do you just have deadlock detection logging turned on, or are you running a full debug kit ? If you're running a full debug kit, there should be a WGDebug.log file as well ...

With regards to IIS, when WG is not bound to any adapter and IIS is started - run a netstat -an. Is IIS listed as "0.0.0.0:80 LISTENING" ?

It's entirely possible that IIS is bound to all adapters, check it's configuration to see if you can only enable it for say, the external adapter, then have WG only on the internal adapter ?

[Bill.Bowen]

It's been awhile, been real busy, but I did solve my problem. Canned the WWW Proxy on port 80. Used GPO to point all users to a WWW Proxy at 8080. Reboots now with no conflicts. System stability vastly improved when I minimized all services logging (none on DNS, WRP, GDP, DHCP, WINS, Scheduler), decreased the History file size to 10000, and kept the cache at 100M or less.