WinGate 9.1.5 become really unstable on Server 2012 or 2016

Use this forum to post questions relating to WinGate, feature requests, technical or configuration problems

Moderator: Qbik Staff

WinGate 9.1.5 become really unstable on Server 2012 or 2016

Postby SpaceJelly » May 30 18 5:54 am

Hi there,

Over the last few months we've noticed Wingate has become really unstable with the underlying OS either no longer passing traffic or the box simply just blue screening with a driver error.

This is not just on one box either, it's across three different boxes that we have deployed with very similar setups.

Brief overview of the builds.

2 x Server 2012 R2 running on vmware esxi 6.5 U2 with the E1000E network cards
1 x Server 2016 running on vmware esxi 6.5 U2 with the E1000E network cards

The Server 2016 is a fresh build to try to solve the issue with crashing however this one is worse and blue screens randomly several times a day!
The blue screens are all to do with the network stack and are either:
Driver IRQL not less or equal on driver e1i63x64.sys
Attempted write to readonly memory on driver tcpip.sys

There's nothing special about the setups. Two NIC's one outside with a public address and gateway, one inside with just the LAN ip and mask. They do have a few static routes for reachability of other internal networks, but that's fairly normal.

It all seems to have started in the last round of windows updates that have occurred over the last few months specifically to fix this RDP issue with CredSSL. However that can't be the main cause, it's all to do with the driver stack.

Why are we not using the VMXNET3 driver? Well, I have tried and the reason is performance! I have never been able to get the speed through Wingate and the VMXNET3 driver over around 20Mb down and 180-200Mb up. With the E1000E driver we easily get 800-900Mb each way (1Gb connection to the internet)

For experimentation I have built another Server 2016 box with the VMXNET3 drivers to try the various tweaks that are around but so far, speeds are as above. Shocking! And since building this box a few hours ago and installing a fresh install of Wingate I'm already getting blue screens!

This is a pretty serious problem as the software is now no longer stable and we cannot use it and have been forced to disable content filtering on one site already due to the constant crashing.

Incidentally, one box that the new Server 2016 install replaced has had Wingate un-installed as well as the Wingate Network Driver and it's now completely stable again and has been up since yesterday evening (early days, but before this box was crashing a few times a day!)

Any thoughts on this issue? If you need access to one of these boxes, please say as RDP or direct console access can be arranged.

Thanks in advance.
SpaceJelly
 
Posts: 11
Joined: Dec 02 15 6:52 am

Re: Wingate 9.1.5 has become really unstable on Server 2012

Postby SpaceJelly » May 31 18 6:11 am

Update: Subject should end or 2016!

Also, it does seem to be how the Wingate driver is interacting with the virtual driver. Each time the box crashes or loses network connectivity you see in the logs loads of Network disconnect then the subsequent reconnect over and over before the box either blue screens or locks up.

This post describes the problem perfectly: https://kb.vmware.com/s/article/2109922

However we're actually using ESXi 6.5 which is not mentioned! Also one machine is Server 2016.

Now, post here: https://communities.vmware.com/thread/4 ... 0&tstart=0 someone states:
Receiving similar error in Server 2012 and 2012r2. we've found that e1000e works more efficiently than e1000 for server 2012, but still causes disconnect issues with server2012r2



For compatibility reasons, we cannot use the vmxnet adapter for compatibility reasons. We found the following workaround caused Event ID 27 to subside:

In Command Prompt

netsh int tcp set global chimney=disabled
netsh int tcp set global rss=disabled
netsh int tcp set global netdma=disabled



From network-adapter properties > Advanced settings, Disable following TCP-offloading options:

-IPv4 Checksum Offload
-Large Send Offload V2 (IPv4)
-Large Send Offload V2 (IPv6)
-TCP Checksum Offload (IPv4)
-TCP Checksum Offload (IPv6)

Over months, this has yet to cause any problems with our applications and has ceased the errors. This still needs a permanent remediation from VMWare though.


I've applied these setting and on the server 2012 r2 boxes they've been ok for the last several hours but still early days.

All the posts I've come across however simply state use the vmxnet3 driver as this is far more stable than the E1000E emulated one. So, how many people actually run Wingate in a VM environment and what network drivers do you use? If vmxnet3 then what's your performance like? As we struggle with the speed, it's shocking!
SpaceJelly
 
Posts: 11
Joined: Dec 02 15 6:52 am

Re: Wingate 9.1.5 has become really unstable on Server 2012

Postby LGM » Jun 09 18 6:05 am

Have you had any luck resolving this?
I am also running Wingate 9.1.5 on Server 2012 and began experiencing multiple daily crashes and blue screening with a driver error the last week of May 2018.
LGM
 
Posts: 1
Joined: Jun 09 18 5:59 am

Re: WinGate 9.1.5 become really unstable on Server 2012 or 2

Postby adrien » Jun 13 18 7:32 am

We've seen a few customers having issue with the vmxnet3 drivers which were resolved by using the supposedly slower E1000 drivers.

As for the bluescreens, did they start happening after a Windows update, or any changes to the WInGate configuration? We see sometimes problems with WinGate's bandwidth control if that is enabled on 2k12. It causes use of a lot more pending sends, which uses a lot more packet buffers (these can be increased in the WinGate advanced options - WGOptions.exe app in the WinGate folder).

Regards

Adrien de Croy
adrien
Qbik Staff
 
Posts: 5441
Joined: Sep 03 03 2:54 pm
Location: Auckland

Re: WinGate 9.1.5 become really unstable on Server 2012 or 2

Postby SpaceJelly » Jun 18 18 10:01 pm

ok, In my experience the vmxnet3 drivers are the slower ones!

Performance (Upload and download speeds) is shocking in Server 2012 R2 with these and wingate.

No wingate, performance is great with these drivers, add wingate and the associated driver to the network stack and performance is around 10% of what it should be.

So, using the E1000 emulated cards.

Also, I have noticed that the problem seems to be more pronounced with NAT usage (IE Transparent, forwarding port 80 and 443 traffic from another firewall to the wingate box to capture non proxy setting users)

If you configure everyone with proxy server settings it seems to be more stable, but without this, and have wingate NAT everything then it gets unstable.

As for the blue screens, they're random and sometimes it doesn't happen, the box just loses visibility of the network. The card is still there, it's just not passing any traffic at all. It's got to the point that I've installed a third party piece of software that monitors the gateway IP and the inside IP and if it stops responding then the software reboots the box! This is a bandaid fix, but at least it keeps the customer working (They have two wingate installs in round robin mode)

Bandwidth control is not enabled, the only things ticked in the network settings page are the Wingate network driver and IPv4 !
SpaceJelly
 
Posts: 11
Joined: Dec 02 15 6:52 am

Re: WinGate 9.1.5 become really unstable on Server 2012 or 2

Postby ggvc » Jun 23 18 12:41 pm

We had similar issues about a year ago. Spoke with Wingate tech support, who recommended switching the VM network driver to VMXNET3 and disabling the Wingate network driver. We did both and issues disappeared immediately.
ggvc
 
Posts: 1
Joined: Jun 23 18 12:38 pm


Return to WinGate

Who is online

Users browsing this forum: No registered users and 30 guests