Bug in SSL implementation

[boblowski]

I think I hit a bug in WinGate's SSL implementation.

I am testing the reverse proxy/WWW functionality at the moment. Test setup: Client (WinXP) <-> WinGate Machine (Win Server 2003 + WinGate) <-> Local webserver. Everything is connected through gigabit and two independent smart switches. The WinGate machine only runs WinGate, with all services except the WWW Service and the Remote Control Service switched off. Windows Firewall and all other relevant Windows services are also switched off. WinGate acts as a reverse proxy for a single webapp.

Without SSL (normal HTTP trafic) everything works perfect. WinGate is rock solid, even under full load / 100% CPU with multiple connections, and the throughput is, even on my somewhat slow test server, excellent.

With SSL/HTTPS throughput is of course lower, but everything is still rock solid, even at full load. There is one specific situations though, where I consistently have problems, and that is with big (>10MB) SSL/HTTPS uploads from a single client through WinGate to the server. Downloads are not a problem, multiple simultaneous connections uploading big files also not.

Only when a single connection to WinGate is sending a big file over SSL to the server and the WinGate computer is (thanks to the gigabit connection) at 100% CPU, then somewhere after between 10 and 13 MB (but almost always after an exact multiple of 1024 bytes) the WinGate.exe process stays at 100% CPU but stops transmitting any packets. Even if I kick the connection, the process stays at 100%. Only when a new connection is established, then after 3-4 seconds everything is back to normal.

* WinGate test computer is a single Pentium III machine and must be at or near 100% CPU. If I limit the bandwidth of the switch between client and WinGate computer to keep CPU below say 90%, the error will not show up.
* A single SSL connection must be uploading. As soon as there are multiple SSL connections uploading, the error will not show up.
* If the uploaded file is less than approx. 10MB in size, there is also no error and everything works as expected.
* The error only shows up with SSL traffic from the client to the reverse proxied server. Downloads and non SSL traffic are not affected.
* ENS driver loaded or not loaded doesn't make a difference. Bandwidth control is switched off.

This error sounds somewhat similar to the one described in forums.qbik.com/viewtopic.php?t=5199, but the solutions in that topic have no effect.

On a faster production machine this error probably won't show up. But I like to be sure about these things :-), so any input is appreciated.

Thanks, Bob

[adrien]

Hi Bob

That's a pretty odd symptom.

We have seen in the past in many cases windows losing thread messages if you pump too many to a single thread at once, I'm suspicious this may be happening, and that having more than one connection alleviates this.

What is the actual data throughput like? I'm trying to get an idea about numbers of thread messages notifying available data.

One comment - if you're piping all inbound connections to the same internal server, you could use a TCP mapping instead of HTTP proxy for reverse proxy. It should give better performance, since it doesn't analyse protocol (as much). You won't get auth with it though, but if all you want is an SSL front-end, it will do that.

Adrien

[boblowski]

Hi Adrien

Thanks for your reply.

We have seen in the past in many cases windows losing thread messages if you pump too many to a single thread at once, I'm suspicious this may be happening, and that having more than one connection alleviates this.

I suspect something like this is happening here as well. For my first test setup I used, like I wrote, a single CPU Pentium III machine. This morning I installed WinGate on a dual Xeon machine (also gigabit+Windows Server 2003), and I got exactly the same symptom. The only difference is that now I can upload upto like 60MB before WinGate stops transmitting packets, and the connection to the client is dropped almost directly, instead of being kept alive.

(BTW, if I do not open a new connection from the client to WinGate, the WinGate process stays in this state forever. The only other way to reset WinGate, is by restarting the Qbik WinGate Engine service.)

As I said, I can alleviate the problem by throttling the incoming bandwidth to port 443, but I would prefer a real solution.

What is the actual data throughput like? I'm trying to get an idea about numbers of thread messages notifying available data.

On the PIII machine I got both down and up around 7MB/s for HTTP and around 5-6MB/s for SSL/HTTPS. (Which I think is really very decent.)

One comment - if you're piping all inbound connections to the same internal server, you could use a TCP mapping instead of HTTP proxy for reverse proxy. It should give better performance, since it doesn't analyse protocol (as much). You won't get auth with it though, but if all you want is an SSL front-end, it will do that.

Thanks for the information. It is nice to have options. Apart from this problem WinGate is really a very solid and flexibel product. Have you ever considered a dedicated reverse proxy solution? Basically the WWW service from WinGate, but then with some goodies like load balancing, custom error pages and URL rewriting?

Thanks, Bob

[adrien]

Hi Bob

Actually the 100% CPU thing makes me think it might be something else.

I"ll see if I can get the test lab to repro the problem. If not, would you be willing to try a test build to track down the issue?

As for the other features... you're describing WinGate 2008. We're close to a beta of that so if you'd like to try it, let me know.

Regards

Adrien

[adrien]

p.s, what web server software is WinGate reverse-proxying to?

[adrien]

we tried to repro this in the lab but didn't have any luck - uploads up to ~70MB worked fine through a fairly slow machine.

Is the uploading agent a web browser, or some other custom software?

Just trying to see if there's anything obvious we may have missed in our test setup.

Regards

Adrien

[boblowski]

Hi Adrien

Thanks for your reply. Sorry to only get back to you now, kind of a busy week over here.

p.s, what web server software is WinGate reverse-proxying to?

It is meant to be used with an in-house developed webapp for customer submission of files to our workflow. For the second test (the one with the Dual Xeon machine for WinGate) I used a program called 'HTTP File Server' as server instead. (Google for it. As far as I know, this program uses the same upload mechanism as our webapp.)

we tried to repro this in the lab but didn't have any luck - uploads up to ~70MB worked fine through a fairly slow machine.

Is the uploading agent a web browser, or some other custom software?

Just a plain webbrowser (FireFox/IE/Safari).

Actually the 100% CPU thing makes me think it might be something else.

I"ll see if I can get the test lab to repro the problem. If not, would you be willing to try a test build to track down the issue?

Sure. Tell me what you had in mind.

As for the other features... you're describing WinGate 2008. We're close to a beta of that so if you'd like to try it, let me know.

That does sound very promising indeed. I'd love to give it a try.

Thanks for your answers, Bob