Proxy PAC file - failover not reliable if WinGate issue

[ik8sqi]

Hi,

We use 3 Wingate proxy servers and provide our users with a PAC file so they can automatically failover to the next working Wingate server should the first go offline. This works great if the actual Wingate server(s) go offline of the Wingate service crashes. But today we had an issues with the primary Wingate server not being able to authenticate users for some reason, so it started to return this error:

Proxy problem
----------------------------------
WinGate does not have a valid user database configured, or it is not available. No web browsing is possible, please contact your system administrator.


We fixed it now (restart of the Wingate service...) so we can't be 100% sure, but it's likely this caused a 504 HTTP error being returned to the browser based on this wingate log entry:
514235323 2/7/2022 8:30:18.442 192.168.24.243 WWW Proxy Server 35772 1801818896 info 0 responding with code 504 Proxy problem


The problem is that the client browser did not interpret the Wingate error as an actual issue that would have triggered the failover to the next server in the proxy.PAC file. So instead of going to the next Wingate server, the browsers displayed that error page to the users.

Our (redacted) relevant section of our PAC file is below. Is there anything we can do either in the PAC file itself or in Wingate's configuration to make it behave so that **Wingate** HTTP 5xx errors cause the next proxy in the PAC to be used? The trick here is that this should only happen for Wingate errors. If a destination website on the internet is having issues and is reporting HTTP 500 errors, well that's an issue with the website and the proxy servers should not be rolled because of that.

Thanks!

Roberto

function FindProxyForURL(url, host)
{
return "PROXY 10.252.1.1:3128; " +
"PROXY 10.252.1.2:3128" +
"PROXY 10.252.1.3:3128";
}

[adrien]

Hi

you'll get that error if using the Active Directory user database connector, if the AD domain controller becomes unavailable.

I would expect in such cases that all the proxies would be affected, unless it's something in the underlying OS.

504 errors are usually only used by gateways, so not common out on the internet. But they are sometimes seen on websites behind reverse proxies.

I don't know if you can switch behaviour in the PAC file based on the error code.

Whenever there's a problem initializing the AD connector, a report is notified, it may be possible to hook even processing to this to automatically reboot the server. However this could be prone to reboot loops if for example the initialization fails first time every time (can happen) and was relying on retry to initialise.

The report posts an incident called "Active Directory Connector initialisation failure", and this will use the default plan, so you will get events processed out of that plan which you can check the incident, and if it matches, then do whatever processing you need. You can possibly stop the WWW proxy service instead of rebooting.

[adrien]

The thing you would do, is inside the event handler for the default plan (e.g. Notify Admin) you would check Incident.Name to see if it is "Active Directory Connector initialisation failure". If so you can use a service control item to stop the WWW proxy service.

Depending on the settings for the notification plan, you'll get notified also in this policy when the incident is closed (e.g. successfully connects to the AD). In that case you could turn the WWW proxy back on.

I've attached a zip file with a sample policy for this.

[ik8sqi]

Thanks Adrien, we're going to give it a try. The issue did affect only our primary WinGate server - the 2nd and 3rd were authenticating correctly while the primary was failing, all are physical (not virtual) servers on the same internal subnet so it seemed to be strictly an issue with Wingate itself which was resolved via a server reboot. It was a sever production issue so we didn't bother trying to restart the qbik service first and went straight to the reboot.

To test your solution we'll apply the policy to our tertiary server, so as to not impact procution and then change the host file on the server to override DNS and change the IP addresses of our domain controllers. This should prevent Wingate from authenticating and hopefully get similar results as to whatever was the original issue with our primary. I'll update the post with the results.

Roberto