Memory problems with qbikhkxp.sys ??

[ErwinB]

For several years now we are having stability issues on our two wingate servers. One is a windows server 2003 machine, the other is running XP sp3, the wingate version used is 6.6.4 build 1338.

The problem is that after a varying number of days, mostly about a week, first the proxy stops responding and some time after that (one, two hours) NAT also stops. If I try to enter gatekeeper I get an error message "out of buffers". At this point I have to restart the machine to get it working again.
If nobody notices the problem, like at night or the weekends, and more time passes then other applications start to fail and eventually the whole computer freezes. A hard reset is then the only solution.
When this happens the event log shows windows errors like "The server was unable to allocate from the system nonpaged pool because the pool was empty." in the time leading up to the freeze.

Using the debugger from the debugging tools for windows I think I was able to determine that the problem is the qbikhkxp.sys driver that is taking up all nonpaged memory.
Using the debugger I found a driver with the identifier BNEG which I think is the internal name for qbikhkxp.sys. Now after two and a half days the debugger shows 857194 Allocs, 490234 frees, a difference of 366960 and used memory 121.133.224 bytes.
As the maximum nonpaged memory on a 32 bit system is 256 megs I have about 2 more days before the trouble begins.
After a restart the memory usage is normal, low. But with usage it creeps up etc.

Do you have any ideas on how to solve this issue?
I checked all services in gatekeeper and made sure they have a time-out because I guess the problem is that buffers are not or not always released.

Thanks in advance,
Erwin

[adrien]

Hi Erwin

Thanks heaps for looking into this.

I'm presuming that BNEG was the pool allocator tag.

Looks like it's our driver all right. I'll need to look into where we could be leaking. Most of our allocs return memory from our own pools (e.g. packet buffers, hash entries etc), so can't leak, so that should narrow it down a bit.

Is there anything significant about the type of network traffic you see on your network? E.g. lots of broadcast traffic or arp?

Do you know how big the allocs are? That would be a great help.

We don't see leaks here - WinGate runs for months without chewing kernel non-paged pool, so I'm guessing it's something to do with some type of traffic that we don't see here that is taking the driver through a code-path that leaks. Or some other issue (perhaps some network interface that's not working properly).

Regards

Adrien

[ErwinB]

Hi Adrien,

Thanks for the quick response.

Unfortunately, I am not an expert in this kind of stuff.
I know that there is a high level of broadcasts on our network because there where some problems with this when our voip system was installed last year.
I am going to give the packet capture feature a try to find out.

Can you maybe tell me how to check the size of the allocs?

Erwin

[ErwinB]

I just made a couple of captures. the number of broadcasts is about 50 a minute with just a couple more arp requests.
Would it be helpful to sent you these captures? If so, where do I sent them to?

Erwin

[adrien]

Hi Erwin

please send the capture to adrien at qbik dot com

thanks

Adrien

[adrien]

Hi

thanks for the captures. It looks like the rate of broadcast is a lot higher, more like 20/s.

Anyway, I'm not sure how to see alloc sizes when monitoring kernel heap sorry. I'll keep looking for an obvious leak, I think we're narrowing it down a fair bit.

Looks like we have some work to do on driver debug logging as well - not much to see in there!

Adrien