Troubles after upgrade

[pgr]

Hi

after upgrading to the latest version a few days ago, I started having troubles with my Wingate server. Two or three times a day I get complaints that "the Internet is not working", I go check the server and I find it is completely hogged, swapping to disk like crazy (it's not a very powerful machine, but anyway it's always been enough and the server doesn't have much activity anyway).

I cannot connect Gatekeeper to see what's going on, so I open up task manager and I find Wingate consuming large amounts of memory (not CPU) and enormous amounts of open handles (over a million sometimes). My only possible way out of this is killing the wingate.exe process and restarting the engine.

This is a Windows XP SP3 machine, running Wingate 6.6.3 and Puresight plugin.

Any advice?
Thanks.

[pgr]

Some more clues...

It seems this is related to one of my users picking up a virus that opens internet connections VERY aggressively, hundreds, thousands. I was lucky enough to watch this happen when I was sitting at the server - he connected, the virus started acting, and Wingate was drowned in a matter of one or two minutes.

Now, I believe I have had users with this kind of virus before, but Wingate seemed to handle things more gracefully - maybe some slow down, but no real vulnerability to this virus action. Is this a bigger and badder virus or is the new Wingate somehow more vulnerable to massive amounts of connections?

Is there any limit built into Wingate to stop accepting connection requests after a certain number, or when the server is under excessive stress? Should there be?

Thanks

[adrien]

Hi

were most of the connections NAT connections, or intercepted by a proxy in WinGate? Intercepting places more load on the server.

I can't think that WinGate should have really changed in how it copes with a huge number of connections recently, so it's more likely that this virus was more aggressive.

What did the traffic look like do you remember?

Regards

Adrien

[pgr]

Hello Adrien


I'm afraid I can't recall what the traffic looked like... I do see some lines in the WWW Proxy Server logs from that user connecting to some sites with very cryptic names... but the logs are no good once the server starts to drown in requests, they stop showing anything.

Is there somewhere I can check my configuration to answer your question? Looking at client activity right now, the lines I see in gatekeeper belong to "WWW Proxy Server Service", I think that where traffic is going through.

I don't know much about this sort of software (as you can see), but I ask: does Wingate enforce any connection limit, or should it?

Thanks

[adrien]

Hi

WinGate doesn't enforce a connection limit, but you can limit bandwidth usage. I don't know if that would help in this case though.

If you still have log files from when the problem occured, you should be able to tell whether it was WWW traffic, or NAT - it will show in the WinGate NAT or WWW Proxy log files.

Regards

Adrien

[logan]

You said that your WinGate server was running Windows XP SP3? If so, it's most likely your operating system that's enforcing a limit on the amount of connections that can be made.

As of XP SP2, Microsoft limited the amount of allowed half-open connections to 10 per second in an attempt to slow down the spread of viruses over the internet. Unfortuantely, the flip side of this change is that it also limits legitimate networking applications that need to make that many or more connections such as WinGate. 10 is simply to low and will degrade the performance of WinGate when under high load. In comparison, the previous half-open connection limit was 65,000.

Check your event viewer for events with the id 4226. These events occur when the 10 half-open connection limit is saturated, and subsequent connections had have been dropped as a result.

It is possible to patch the TCPIP.sys in order to raise the half-open connetion limit back to 65,000 and improve the performance of WinGate if this is a problem. Searching google for "how to patch tcpip.sys" nets many results on how to do this, so I will let you search this for yourself rather than repeat already abundant information.

[pgr]

The problem is still ocurring (the user still has the virus, and I'm keeping it so I can troubleshoot this thoroughly.

I found that the Winsock Redirector Log was the one that had become huge after one of the server freezes happened today.

It has over 300 entries per second that look like this:
09/18/09 16:23:00 192.168.0.5 user123 124210 Error: Request 278 connect error 10049
09/18/09 16:23:00 192.168.0.5 user123 124316 Error: Request 277 connect error 10049
09/18/09 16:23:00 192.168.0.5 user123 124351 Error: Request 279 connect error 10049

@logan: I am aware of the SP2 issue, but I think I have the inverse problem - I don't need more connections, I need less! :-)

I would like to get to the bottom of this. If you think it's advisable, I will open a Support ticket and that way I can send you my configuration, logs, etc.

Thanks

[adrien]

error 10049 is usually caused by DNS lookup failures, where something does a lookup, and tries to connect to that IP without checking the result of the lookup.

Since the WRP server relies on the client to specify an IP to connect to, that implies the client is losing DNS somehow. Is that client using WinGate as a DNS server?

Regards

Adrien

[pgr]

Yes, the client should be using Wingate for DNS.

I checked the DNS Service log and there many entries there like this

09/18/09 16:22:02 192.168.0.5 user123 0000124764 Requested: DNS: A lookup "10.smo7he.com."
09/18/09 16:22:02 192.168.0.5 user123 0000124765 Requested: DNS: A lookup "10.smo7he.com."
09/18/09 16:22:02 192.168.0.5 user123 0000124766 Requested: DNS: A lookup "10.smo7he.com."

Always that same address. But these entries happen only about 10 times per second, not hundreds like on the other log.

[logan]

That DNS service log is slightly suspect. A client should only really need to to a DNS lookup for a domain name once, but in the log snippet that you posted, it was the same client asking for the same DNS address over three different sessions. This could indicate that the client isn't getting what it wants from the DNS lookups. It's hard to tell based on standard log files though. This only tells us that the client made a request which in itself doesn't mean anything. Can you do the following? Thanks.

1. Enable debug logging on the effected services (so far: WRP service, DNS Service, DNS/WINS Resolver)..
2. Let the problem happen again, then check the resulting log files. I think we will be particularly interested in the DNS Resolver log.

[logan]

If you could send a full copy of the logs to support@qbik.com as well, that would be helpful.

[pgr]

Just an update on this -

I'm afraid my dear user managed to erradicate the virus before I was able to capture the debug logs, so now I can't really help with the investigation of this problem.

If it happens again I promise to send more information, I'll know as soon as it happens - it's a very noticeable problem.

Thanks.