Wingate is blocking packets passed by border Cisco router

[Alen]

Hi,

Our connection to Internet is made this way:
LAN - Core router - Wingate - Border router.

I have a strange situation, on the border router, which is Cisco 871 I have ACL on external interfaces for incoming traffic ("deny ip any any") plus I have stateful firewall enabled there (ip inspect for icmp, udp, tcp and even for http, https, ftp, pop3, smtp - all protos for which I use Wingate proxy services). The router is also making NAT for Wingate.
Wingate is used mostly as proxy (only 3-4 NAT clients, but I checked the issue also when the NAT clients were inactive).

Now, the strange thing is every day ~ 200-300 packets are blocked by Wingate firewall with source ip, some of which belong to the well known Internet servers (I checked a few) and source ports 80 and 443 (almost all packets. Sometimes I also see some packets from other ports).

It looks like the border router for some reason passes packets as authorized and Wingate does not.
IMHO, it could happen either if packets are recognised as unauthorised for Wingate or because Cisco for some reason "corrupts" or tranfers already corrupted packets to Wingate. I was thinking about corruption and checked all variants I suspected:
- Most probable reason is packet fragmentation - I have enabled "ip virtual-reassembly" on all interfaces - external and internal and it doesn't help. (By default when NAT is activated Cisco activates also reassembly on respective interface.)
- I also found an old post stating the problem could be because of the Cisco's CEF switching - disabling CEF also did not solve the problem.
- Disabling routers firewall closes Internet connection at all => the routers ACL is working as has to.

I need help to figure out the reason.
Again: either packets are (indeed or just looked like) corrupted for Wingate or they are not recognized by it as authorised (i.e. replies on our requests) packets.

I am ready to provide all necessary data. I am attaching the router config and connection schema.

P.S. This is critical, because every day we have to make some procedures stated in the appropriate policies and write explanations (meanwhile we do not know the exact reason of the issue).

Attachments

IBRConfig.zip

(1.74 KiB) Downloaded 333 times

ConnectionTopology.gif (10.82 KiB) Viewed 13339 times

[adrien]

Hi

check the TCP flags for these blocked packets. The usual scenario for this is when a connection is closed but a late RST packet is received, which then shows as blocked.

Regards

Adrien

[Alen]

Here are 4 latest blocked packets report:

Wingate firewall hit report:

Time: 14/07/10 10:10:00
Reason: Port Range
Source MAC address: 00-26-0B-6B-B1-73
Destination MAC address: 00-10-B5-F8-13-DD
Source IP Address: 217.73.200.219 : 80
Destination IP Address: 192.168.200.41 : 3976
Protocol: TCP
TCP flags: R
Time-to-live: 254


Wingate firewall hit report:

Time: 14/07/10 10:12:18
Reason: Port Range
Source MAC address: 00-26-0B-6B-B1-73
Destination MAC address: 00-10-B5-F8-13-DD
Source IP Address: 198.133.219.10 : 443
Destination IP Address: 192.168.200.41 : 2384
Protocol: TCP
TCP flags: R
Time-to-live: 254


Wingate firewall hit report:

Time: 14/07/10 10:12:22
Reason: Port Range
Source MAC address: 00-26-0B-6B-B1-73
Destination MAC address: 00-10-B5-F8-13-DD
Source IP Address: 72.163.4.161 : 443
Destination IP Address: 192.168.200.41 : 2343
Protocol: TCP
TCP flags: R
Time-to-live: 225


Wingate firewall hit report:

Time: 14/07/10 10:13:05
Reason: Port Range
Source MAC address: 00-26-0B-6B-B1-73
Destination MAC address: 00-10-B5-F8-13-DD
Source IP Address: 217.212.252.192 : 80
Destination IP Address: 192.168.200.41 : 2362
Protocol: TCP
TCP flags: R
Time-to-live: 254

Obviously "TCP flags: R" is what you are saying.
Very well, finally I have an explanation.

Now, can we somehow remove this packets from the blocked list or (which is much correct) make Cisco to drop them?
Being more precise, what is the default TCP connection timeout (I'll try to find a way to set it {minus 2-3sec} on the border router)?

P.S. Thank you, Adrien.

[Alen]

Adrien, I mentioned some packets have "RA" flag. What does it mean?

Wingate firewall hit report:

Time: 14/07/10 18:38:36
Reason: Port Range
Source MAC address: 00-26-0B-6B-B1-73
Destination MAC address: 00-10-B5-F8-13-DD
Source IP Address: 195.222.17.41 : 80
Destination IP Address: 192.168.200.41 : 2195
Protocol: TCP
TCP flags: RA
Time-to-live: 40

[logan]

A refers to the ACK flag (acknowledge)
R refers to the RST flag (reset)

An RA packet is sent when an R packet is received. You will see an RA packet being blocked for the same reason as an R packet. The session in the firewall has already timed out or been closed by the firewall, so the packet isn't associated with any open sessions and is dropped.

Now, can we somehow remove this packets from the blocked list or (which is much correct) make Cisco to drop them?

Well, these blocked packets aren't really in error, so WinGate is correct to record them. I haven't used WG6 for a while, but I'll have a look later today and see if you can disable these warnings.

I know that juniper routers have an option called "TCP RST Invalidate Session Immediately" which causes sessions to be closed immediately when an RST packet is received. It sounds like something similar might be enabled on the path of these sessions (probably on the remote side if WinGate is getting the warnings). If your local Cisco has an option like this, try toggling it, which may cause it to deal with extraneous R / RA packets itself.

Being more precise, what is the default TCP connection timeout (I'll try to find a way to set it {minus 2-3sec} on the border router)?

Hmmm, that changes from router to router, sorry.

[Alen]

Thank you, Logan, for the clarification.

Well, these blocked packets aren't really in error, so WinGate is correct to record them. I haven't used WG6 for a while, but I'll have a look later today and see if you can disable these warnings.

Please, be so kind.

I know that juniper routers have an option called "TCP RST Invalidate Session Immediately" which causes sessions to be closed immediately when an RST packet is received. It sounds like something similar might be enabled on the path of these sessions (probably on the remote side if WinGate is getting the warnings). If your local Cisco has an option like this, try toggling it, which may cause it to deal with extraneous R / RA packets itself.

I'll ask about such possibility on Cisco forum, thank you.

Last edited by Alen on Jul 22 10 5:01 pm, edited 1 time in total.

[Alen]

I know that juniper routers have an option called "TCP RST Invalidate Session Immediately" which causes sessions to be closed immediately when an RST packet is received. It sounds like something similar might be enabled on the path of these sessions (probably on the remote side if WinGate is getting the warnings). If your local Cisco has an option like this, try toggling it, which may cause it to deal with extraneous R / RA packets itself.

I asked on Cisco forum and was advised to try to set "ip nat translation finrst-timeout" to the minimum value (which is 1 second). I tried it and it does not help.
Please help.

[Alen]

Logan, yesterday I understand that the best variant would be to increase wait time for R and RA packets of closed sessions in Wingate (or OS?). Is it possible?

[logan]

Logan, yesterday I understand that the best variant would be to increase wait time for R and RA packets of closed sessions in Wingate (or OS?). Is it possible?

You can modify a few of the timeout values from the Advanced Options applet pictured below

Start -> Programs -> WinGate -> Advanced Options

WinGate 6/Advanced Options/Extended Networking - Timeouts

wg6-advopts-timeouts.jpg (65.5 KiB) Viewed 13227 times

[Alen]

Ok, thank you. But do you have any recommendations about the values?
And do you expect that will help?

[logan]

I haven't tested the effects of these options, so I can't make a recommendation. Try increasing the TCP time-out value in 10 second increments and then monitor for a while after each change to note any effects.

The driver needs to be reloaded to pick up the changes, so the computer must be restarted after making any changes.

[Alen]

I increased TCP timeout value up to 3600 sec, still no success. I see a lot of blocked TCP packets with R flag.
And the strange thing is after the timeout last increase and Wingate restart, blocked packets started to appear in the firewall tab earlier than after 1 hour!?
IMHO, this could mean the problem has another reason...

What to do now?

[logan]

Having a 1 second timeout on the Cisco may still be long enough to let these packets through, as packets usually take only milliseconds to travel between hosts. I assume that I was wrong about the TCP timeout setting in the advanced options applet, in that it doesn't effect WinGate's immediate session invalidation (which the more I look into, seems to be hard coded). The more I think about it, the more it seems that the only way this can really be resolved is to get the Cisco to invalidate the sessions immediately, and therefore block the packets responding to a closed session itself.

I noted from your first post that you check WinGate's firewall daily as part of your security policy/practice, so I'm guessing that you want to hide these packets to reduce the amount of insignificant noise in the firewall tab. That being the case, there may be an alternate solution to filtering the noise. Instead of reading the packets in the firewall tab, you could read them from the actual log file. The only hurdle is that the information about blocked packets is encoded so not very useful without first being decoded.

All that would be needed is a program that can parse the firewall log file, decode the blocked packet information, and then filter out the packets that you don't want to see. That's the tricky part.

[Alen]

The more I think about it, the more it seems that the only way this can really be resolved is to get the Cisco to invalidate the sessions immediately, and therefore block the packets responding to a closed session itself.

Unfortunately, I failed to provide it. And I don't know any other way, except "ip nat translation finrst-timeout"...

Do you have the same behaviour in Wingate 7?

I noted from your first post that you check WinGate's firewall daily as part of your security policy/practice, so I'm guessing that you want to hide these packets to reduce the amount of insignificant noise in the firewall tab.

Yes.

That being the case, there may be an alternate solution to filtering the noise. Instead of reading the packets in the firewall tab, you could read them from the actual log file. The only hurdle is that the information about blocked packets is encoded so not very useful without first being decoded.
All that would be needed is a program that can parse the firewall log file, decode the blocked packet information, and then filter out the packets that you don't want to see. That's the tricky part.

Ok, name the program.

We have IAM for Wingate, but I doubt it has such functionality. I'll look at it and post the info.


P.S. I don't know did the TCP timeout increasing up to 60 minutes help or not (obviously not, as when it was equal to 20 minutes the situation did not change), but it is already a couple of days I see ~ 10 time less blocked packets than previously! I know our provider recently has changed its global provider, may be that helped?!
I'll change Wingate TCP timeout to its defaults and see what happens.

[logan]

I don't know of any programs that would be able to do this, so I'm writing a simple program that will read the log file, decode the actual packet info and then filter out packets with the RST flag or RST and ACK flags. I'll send you a copy when it's working.

[Alen]

I don't know of any programs that would be able to do this, so I'm writing a simple program that will read the log file, decode the actual packet info and then filter out packets with the RST flag or RST and ACK flags. I'll send you a copy when it's working.

And it will be nice if you integrate it into Wingate and allow to setup Wingate to ignore blocked R packets. :-)

[logan]

Sorry that this is taking a while. I don't have much time up my sleeve for on-the-side projects, so I've only just reached the point where I'm pulling real usable data out of that base64 string. All I have to do now is make sense of all these bytes of data and what they represent. Picking out the IP's was easy so that I've done, but I need to consult a developer to figure out the rest.

Attachments

QbikFWLP.png (27.06 KiB) Viewed 13045 times

[Alen]

Hi Adrien, Logan!

Finally I found the reason and source of packets being blocked. You should knew this!

When you have Default action = "Allow" in Port Security for packets from LAN to Internet, it does not matter if a user have any rights for NAT connection. Any software, including viruses can send packets via NAT connection to any Internet server. And only reply packets from the server is blocked by Wingate.

And those were the packets I was seeing in Firewall tab. And the border Cisco was passing them because he saw they are reply packets. So the Cisco is not guilty.


Now explain me my dear developers ;-), why packets are allowed to go out, if the user have no rights for NAT connection?!

[adrien]

Hi

You are correct. When using ENS policy, the policy is applied in the WinGate engine once the engine is notified that there is a new connection made.

If the connection is disallowed by ENS policy, then the connection is terminated.

By this stage, the initial packet has already been forwarded.

we tried several options to get around this, such as deferring processing of the initial packet until a decision as made by the engine, but this was problematic due to issues around the communication channel between the driver and our engine.

We have plans to change this as part of our changes proposed for 7.1 as per http://forum.wingate.com/viewtopic.php?f=12&t=40142#p36796

Regards

Adrien

[Alen]

You are correct. When using ENS policy, the policy is applied in the WinGate engine once the engine is notified that there is a new connection made.
If the connection is disallowed by ENS policy, then the connection is terminated.
By this stage, the initial packet has already been forwarded.

Adrien, this should be fixed. It is inadmissible for a normal firewall.

P.S. I believe Wingate is not for SOHO only, but also for SMB. And you should change\add some options to make it fully suitable for use by SMB.

[adrien]

Hi

One way of looking at it is that there are actually 2 firewalls in WinGate ENS.

There's the port security settings, which you can use to block ports based on type of interface and protocol. This is all evaluated in the driver itself, and so no packets are forwarded until the decision has been made.

The second one is the old WinGate 6 ENS policy. This was mainly intended to control user access to services where the client was using NAT. Since we wanted to provide policy access to data that only existed in the engine (such as user account information etc), we needed the engine to make the decision for the driver. Problems arise when the engine is not available (e.g. stopped), or when the rate of connections overwhelms the bandwidth of the driver <-> engine comms channel. For various historical reasons it was in the end decided to make the engine terminate connections after the fact.

This achieves the main goal of allowing prevention of certain types of user behaviour / use. It was never really meant to be considered a full firewall in terms of security considerations.

Anyway, that's the history of it.

I agree it's not an ideal way to do it. We even prototyped a system which pended the initial packets until a decision had been communicated back from the engine, but this suffered from various issues, including stabilty and performance, and would require a re-architecture of our packet processing systems in the driver in order to be successful.

In the end, we wish to be able to offer policy control over what happens to a connection, e.g. divert it somewhere (replace dest IP / port).

To do this requires making the decision before forwarding the first packet. So we have additional incentives based on desired features which will also address this issue.

Regards

Adrien

[Alen]

I understand.
Hope you will have time, desire and resources to make the planned changes...