WG 6.5.2 All Proxys and Mail server stop passing packets

[kgoodknecht]

Hi all,

I had this problem on 6.2.2, I hoped that upgrading to 6.5.2 would help, but it seems to have made matters worse. The ENS driver is completely disabled so that RRAS can be enabled with several demand dial site to site VPN connections. At varying times, sometimes less than an hour, sometimes 3 to 6 hours, but always the Wingate Proxy and SMTP server (at least) will just stop passing web requests and e-mail. Restarting the Wingate service gets things going again, but within 1-6 hours it all comes to a screeching halt. The KAV 2.0.3 plug in is installed and it is running on a dedicated Win2k3 R2 2.4 GHz w/2 GB of RAM which should be plenty of power to serve 12 users. Is there anything I've missed that I can try to stop this aggravating behavior?

[adrien]

Hi Kevin

Can you still log in with GateKeeper when it stops? We've had some timeout issues in the driver with 6.5.2. We've been working on a new build for a while... latest is at

http://www.wingate.com/downloads/WinGat ... 68-USE.EXE

You could give this a try if you like

Regards

Adrien

[kgoodknecht]

Thank you Adrien, I have installed and will let you know within the next 24 hours if we have a fix.

[kgoodknecht]

Adrien, it almost assuredly has become apparent that 6.5.3 has fixed the problem, it has been 20 hours since I installed it on the server exhibiting the issue the worst, it was lucky to make it 3 hours before, my own server used to lock up at least once or twice a day. I have just upgraded an additional 3 servers all exhibiting the same problem in varying degrees. Thank you very much for getting this to me so quickly, I was having some pretty irate customers over the problem, I just notified them that the problem should be substantially reduced if not eliminated.

[adrien]

Hi

Thanks for that news

We're getting good reports about 6.5.3, but still a couple of problems I fixed today even (with Java auth). We aim to get it released within a week or 2.

Regards

Adrien

[sysneticlabs]

Hi Adrien !

I add my OWN feedback on V6.5.3 => It fix my BOSD and everything is working fine for me

Windows Server 2008 - standard - English + Wingate V6.5.3 Build 1268 + usage of ENS !

best regards
Pascal

[kgoodknecht]

6.5.3 Seems to have fixed the proxy problem, but I'm having a new problem with the SMTP server. Outbound mail stops flowing out, in the Activity pane, it says Connecting to <domainname> Or Connecting to <FQDN of external forwarded Server>. If I set the forwarding servers IP address it goes out without a problem. Inbound mail comes in and gets forwarded to my Exchange server just fine, even by name.

It seems that this must be a DNS resolution problem, since it can send to an IP address and not a name. Restarting the Wingate service fixes it temporarily, very temporarily, because it doesn't last long and mail starts backing up again. If I look in my DNS server cached lookups I see all the MX and A record lookups, so it seems to be something internal in Wingate. I've tried both settings for "Enable lookup of cached resources" and it didn't seem to make a difference.

[adrien]

Hi

We've seen a couple of reports recently about DNS servers stopping responding to WinGate's DNS resolver.

Restarting the engine fixes it, so it seems to be something to do with the client port the resolver uses. It uses the same port for all requests / responses.

Packet captures show responses stop coming, although other responses from the same server to other processes on the WIngate computer show that the DNS server is still responsive.

We are wondering if some recent DNS server release now forces DNS resolvers to not reuse source ports due to DNS cache poisoning issues. It would be unfortunate if that were the
case, since cache poisoning can be addressed without randomising resolver source port.

Either that or some intermediary makes say a UDP connection entry invalid and requests stop making it out to the servers.

Does this happen from any particular version of WinGate onwards?

Regards

Adrien

[zhygavin]

I have encountered the same problem for serveral monthes,but till to now I can get the right answer,I am very disappointed.I want to disgrade to 6.0.3 now.The ticket Id is DLZ-37561.My english is poor,I hope you can understand what I said.

Hi

We've seen a couple of reports recently about DNS servers stopping responding to WinGate's DNS resolver.

Restarting the engine fixes it, so it seems to be something to do with the client port the resolver uses. It uses the same port for all requests / responses.

Packet captures show responses stop coming, although other responses from the same server to other processes on the WIngate computer show that the DNS server is still responsive.

We are wondering if some recent DNS server release now forces DNS resolvers to not reuse source ports due to DNS cache poisoning issues. It would be unfortunate if that were the
case, since cache poisoning can be addressed without randomising resolver source port.

Either that or some intermediary makes say a UDP connection entry invalid and requests stop making it out to the servers.

Does this happen from any particular version of WinGate onwards?

Regards

Adrien

[adrien]

HI

I just read through the ticket.

So, are you saying that 6.0.3 works fine, but 6.5.2 / 6.5.3 do not?

If this is the case, then there is some other problem than the one we have been looking into, since the DNS resolver in 6.5.3 basically behaves the same way as 6.0.3.

I note you did some packet captures. There's one change we did for the resolver, relating to handling of ICMP errors on the resolver socket.

When you logged the packet capture, I think you only logged UDP/53 right? Could you repeat the capture but also log ICMP as well? That will show us if an ICMP dest unreachable error is being sent back to the DNS resolver port.

Hopefully we can solve this soon - if 6.0.3 worked and 6.5.2 doesn't that narrows down the scope considerably, especially since you don't have ENS installed.

Regards

Adrien

[zhygavin]

I don't know if the problem can be solved when I degrade the wingate server to 6.0.3.I have been using the wingate 6.0.3 for years,and I used a new machine which windows server 2003 R2 was installed last year,but the problem appears from this March,serveral times an hour or several times a day.I have replaced the new machine with the old one which has been upgraded to 6.2.2 too,the problem appears too.I upgraded the wingate to 6.5.2 and 6.5.3,but the problem still can not be solved. what can I do ? My manager hope I can solved this problem as quickly as ,The following is my licenseinformation:
WinGate 6.x Professional 100 concurrent users
Licence id: 485595
Registered to: zhpenavico

I don't know if I only log UDP/53 , Could you tell me how to change the settings for logging ICMP ,I did'nt changed any settings after I installd the captured software.

HI

I just read through the ticket.

So, are you saying that 6.0.3 works fine, but 6.5.2 / 6.5.3 do not?

If this is the case, then there is some other problem than the one we have been looking into, since the DNS resolver in 6.5.3 basically behaves the same way as 6.0.3.

I note you did some packet captures. There's one change we did for the resolver, relating to handling of ICMP errors on the resolver socket.

When you logged the packet capture, I think you only logged UDP/53 right? Could you repeat the capture but also log ICMP as well? That will show us if an ICMP dest unreachable error is being sent back to the DNS resolver port.

Hopefully we can solve this soon - if 6.0.3 worked and 6.5.2 doesn't that narrows down the scope considerably, especially since you don't have ENS installed.

Regards

Adrien

[adrien]

Hi

have you checked with your ISP to see if they changed their DNS server software, or patched it about the time you started seeing these problems?

Regards

Adrien

[zhygavin]

Sorry for my english,I don't understand the meaning of the following sentence .
"Either that or some intermediary makes say a UDP connection entry invalid and requests stop making it out to the servers."

Hi

We've seen a couple of reports recently about DNS servers stopping responding to WinGate's DNS resolver.

Restarting the engine fixes it, so it seems to be something to do with the client port the resolver uses. It uses the same port for all requests / responses.

Packet captures show responses stop coming, although other responses from the same server to other processes on the WIngate computer show that the DNS server is still responsive.

We are wondering if some recent DNS server release now forces DNS resolvers to not reuse source ports due to DNS cache poisoning issues. It would be unfortunate if that were the
case, since cache poisoning can be addressed without randomising resolver source port.

Either that or some intermediary makes say a UDP connection entry invalid and requests stop making it out to the servers.

Does this happen from any particular version of WinGate onwards?

Regards

Adrien

[adrien]

If WinGate isn't directly connected to the internet, then DNS requests will be forwarded out to the internet (presumably to your ISP's DNS server) by some intermediary - a device that sits between WinGate and the internet connection.

If that device is dropping the DNS request packets after a while, that would also cause this. If you have such a device, was its firmware upgraded recently, or is it new?

Regards

Adrien

[zhygavin]

I have a netscreen firewall between the wingate and the internet connection,it has been used for years and its firmware didn't upgrade recently. And when the problem appears,the wingate server itself can browse the web pages normally.

If WinGate isn't directly connected to the internet, then DNS requests will be forwarded out to the internet (presumably to your ISP's DNS server) by some intermediary - a device that sits between WinGate and the internet connection.

If that device is dropping the DNS request packets after a while, that would also cause this. If you have such a device, was its firmware upgraded recently, or is it new?

Regards

Adrien

[zhygavin]

I can not get the information from my ISP,because they think It is a secret.do you have other methods to find the cause?

Hi

have you checked with your ISP to see if they changed their DNS server software, or patched it about the time you started seeing these problems?

Regards

Adrien

[adrien]

I'm just wondering if there is some other DNS server you can get WinGate to use.

From the packet captures you sent, it looks like after a while the DNS servers stop responding to WinGate's DNS resolver, but they don't stop responding to other DNS resolvers on the WinGate machine (e.g. the OS resolver, which your browser is using). So, the question is why do they stop responding only to the WinGate resolver?

The packet captures don't show any problem with the composition of the request packets formed by WinGate, so the only other main difference is that WinGate always uses the same socket to send all DNS queries, which means that all the DNS queries will come from the same source port.

The OS resolver on the other hand uses a new socket for every request, so the source port changes for each request.

So there are only a couple of possible options.

a) something between WinGate and your ISP decides it won't forward packets from that source port any more (e.g. the Netscreen)
b) your ISP DNS service decides it won't respond to that source port any more. If you explain your problem to them, they might be able to do something about it.

A quick fix could be as follows.

1. Get your AD DNS server to forward to your ISP DNS Server rather than WinGate. It will then use UDP NAT to do this, but sends each request on a new source port.
2. Get WinGate to use the AD DNS server.

Then the only DNS requests going out of your network will be from the AD DNS server.

Regards

Adrien

[zhygavin]

I understand what you said,but I don't have AD in my LAN,so you think what I can do to solve this problem. I have asked my eseller about this problem , they said that I could use the DNS service in the OS insead of the wingate's DNS service, I have tried it and the problem still appeared .

I'm just wondering if there is some other DNS server you can get WinGate to use.

From the packet captures you sent, it looks like after a while the DNS servers stop responding to WinGate's DNS resolver, but they don't stop responding to other DNS resolvers on the WinGate machine (e.g. the OS resolver, which your browser is using). So, the question is why do they stop responding only to the WinGate resolver?

The packet captures don't show any problem with the composition of the request packets formed by WinGate, so the only other main difference is that WinGate always uses the same socket to send all DNS queries, which means that all the DNS queries will come from the same source port.

The OS resolver on the other hand uses a new socket for every request, so the source port changes for each request.

So there are only a couple of possible options.

a) something between WinGate and your ISP decides it won't forward packets from that source port any more (e.g. the Netscreen)
b) your ISP DNS service decides it won't respond to that source port any more. If you explain your problem to them, they might be able to do something about it.

A quick fix could be as follows.

1. Get your AD DNS server to forward to your ISP DNS Server rather than WinGate. It will then use UDP NAT to do this, but sends each request on a new source port.
2. Get WinGate to use the AD DNS server.

Then the only DNS requests going out of your network will be from the AD DNS server.

Regards

Adrien

[adrien]

do you have any other firewall software operating on the WinGate computer?

If you tried using a DNS server on your LAN, can you get a packet capture on that computer to make sure that the request packets are getting to it?

The problem with taking packet captures from the WinGate machine when looking for outbound (sent) packets, is that other firewall software may block a packet just after the packet capture software sees it, so you can't be certain if the packet went out on the ethernet or not. Capturing from another machine solves this, since if you see it, it must have been sent by the other computer.

Adrien

[zhygavin]

The following is the wingate server's configuration:
external network card:
IP:192.168.2.2
Mask:255.255.255.0
Gateway:192.168.2.1
Primary DNS:202.96.128.166
Secondary DNS:202.96.128.86
internal network card:
IP:192.168.0.86
Mask:255.255.255.0
Gateway:No
Primary DNS:NO
Secondary DNS:No

I want to install another machine to implement the DNS service,how do I configure the machine for the network card and DNS software ,thank you very much for your help.

do you have any other firewall software operating on the WinGate computer?

If you tried using a DNS server on your LAN, can you get a packet capture on that computer to make sure that the request packets are getting to it?

The problem with taking packet captures from the WinGate machine when looking for outbound (sent) packets, is that other firewall software may block a packet just after the packet capture software sees it, so you can't be certain if the packet went out on the ethernet or not. Capturing from another machine solves this, since if you see it, it must have been sent by the other computer.

Adrien

[adrien]

are you going to use MS DNS server or some other?

If MS one, you need to configure the forwarders for it (it's different to the DNS settings for a NIC). Set that to the DNS servers that your WinGate computer is currently using, then set WinGate to use that new DNS server.

Regards

Adrien

[zhygavin]

I will use MS one ,I don't know what IP I sould set,can you tell me ,thank you very much.

are you going to use MS DNS server or some other?

If MS one, you need to configure the forwarders for it (it's different to the DNS settings for a NIC). Set that to the DNS servers that your WinGate computer is currently using, then set WinGate to use that new DNS server.

Regards

Adrien

[adrien]

Hi

Which IP are you referring to? There are several.

1. The IP address that WinGate uses as it's DNS resolver
2. The IP address that the network cards on the WinGate machine use as their DNS resolver
3. The IP address that the DNS server you are installing will use as a forwarder
4. The IP address that the network cards on the new DNS server will use as their DNS resolver?

Normally in this case

1. Empty
2. The IP of the new DNS server
3. the IP addresses that WinGate currently uses (Primary DNS:202.96.128.166, Secondary DNS:202.96.128.86)
4. nomally 127.0.0.1 and this is normally set when you set up the DNS server

Regards

Adrien



Regards

Adrien

[zhygavin]

Thank you very much for your reply, I want to install another machine as a DNS server, I think that It should be able to connnect to the internet ,but all machines in my company connect to the internet by the wingate server,so should I set the new machine to 192.168.0.X or to 192.168.2.X or other IP ?

Hi

Which IP are you referring to? There are several.

1. The IP address that WinGate uses as it's DNS resolver
2. The IP address that the network cards on the WinGate machine use as their DNS resolver
3. The IP address that the DNS server you are installing will use as a forwarder
4. The IP address that the network cards on the new DNS server will use as their DNS resolver?

Normally in this case

1. Empty
2. The IP of the new DNS server
3. the IP addresses that WinGate currently uses (Primary DNS:202.96.128.166, Secondary DNS:202.96.128.86)
4. nomally 127.0.0.1 and this is normally set when you set up the DNS server

Regards

Adrien



Regards

Adrien

[adrien]

Another option is to install the MS DNS server on the WinGate machine itself. That will save you adding another machine.

Then you would disable the DNS service in WinGate, and set the DNS server to use in the WinGate DNS resolver to the Internal IP address of the WinGate computer, so it uses the local DNS server.

That way the rest of your LAN can also use the MS DNS server, which supports more things than the WinGate one does, such as dynamic registration of records etc.

Adrien

[zhygavin]

If I install the MS DNS server on the Wingate machine itself,do I need to set the Wingate DNS resolver to 192.168.0.86? thank you very much.

Another option is to install the MS DNS server on the WinGate machine itself. That will save you adding another machine.

Then you would disable the DNS service in WinGate, and set the DNS server to use in the WinGate DNS resolver to the Internal IP address of the WinGate computer, so it uses the local DNS server.

That way the rest of your LAN can also use the MS DNS server, which supports more things than the WinGate one does, such as dynamic registration of records etc.

Adrien

[adrien]

That's the IP of your internal adapter, so yes that should work.

Adrien

[zhygavin]

Thank you very much for your help , The problem seems to be solved after I Installed the MS DNS in the wingate server itself.

[kgoodknecht]

Hi

We've seen a couple of reports recently about DNS servers stopping responding to WinGate's DNS resolver.

Restarting the engine fixes it, so it seems to be something to do with the client port the resolver uses. It uses the same port for all requests / responses.

Packet captures show responses stop coming, although other responses from the same server to other processes on the WIngate computer show that the DNS server is still responsive.

We are wondering if some recent DNS server release now forces DNS resolvers to not reuse source ports due to DNS cache poisoning issues. It would be unfortunate if that were the
case, since cache poisoning can be addressed without randomising resolver source port.

Either that or some intermediary makes say a UDP connection entry invalid and requests stop making it out to the servers.

Does this happen from any particular version of WinGate onwards?

Regards

Adrien

Hello Adrien,

I would not be surprised if the update you are referring to is causing this since all of my Wingate server also run MS DNS, this update may have fixed one problem, but it opened up a can of worms on many others causing port conflicts. The most serious of these are when DNS latches on the port 4500, which causes IPSec to go in to Block mode, I taken many calls on this one and there's no telling how many people have not made the connection and are still pulling their hair out.

Anyway, to go back to a particular point in time and say when it started is a little difficult, I ran 6.2.2 for a long time and didn't notice it happening, and when it did start I did not make the connection to the DNS update, but I would have to say yes, it started about that time period. This would make it very unfortunate if it were the update since I only manage one Wingate server that isn't running on a DC, and it is a production machine, so it makes it difficult to do a test on a machine with out MS DNS running. I will talk to the owner of that server to see if we can test this.

I apologize for not getting back sooner on this, the past couple of months have been really hectic, and now I've got a whole new set of issues with a new server, and I will be starting a new thread on that one.