WinGate appears to block downloads without notice.

[Randy Baker]

I am evaluating WinGate for Firewall/SiteFiltering/Perimeter AV.

The test configuration includes 7 systems of various O/S's such as OS/X, various Windows versions and various Linux distributions. I have WinGate configured with "User Assumption".

Inside my network, I am attempting to download the latest distribution of Ubuntu from the Ubuntu download website. The download attempts fail due to timeouts. However, I can download this file from the system WinGate is installed on. This appears to be a simple HTTP download.

I cannot determine what is preventing me from downloading from machines behind WinGate.

[adrien]

Hi

Are you using any plugins, such as antivirus scanning?

Can you please enable logging of session termination and debug logging in the WWW proxy?

Then if you try again, can you post the logs you get relating to this request so we can see what's causing the break?

Regards

Adrien

[Randy Baker]

I have the AV plugin, and have enabled logging. There is no log entry suggesting that the AV engine is blocking the download.

If I disable WinGate and restart WinProxy, and using the same workstation as the first attempt, I can get the download "Save as" dialog box within 15 seconds. After disabling WinProxy and restarting WinGate, the behaviour resumes.

In reviewing the logs, it appears to fail quickly. However, the web page does seem to indicate it is retrying for tens of minutes before eventually timing out. There are no entries in the log that indicate that the page is retrying.

The following is from the WWW proxy log, with http removed due to the message limitations.

11/10/07 22:46:37 192.168.1.6 Guest 0000000081 Created:
11/10/07 22:46:37 192.168.1.6 Guest 0000000081 Requested: ubuntu.media.mit.edu/ubuntu-releases/gutsy/ubuntu-7.10-server-amd64.iso
11/10/07 22:46:37 192.168.1.6 Guest 0000000081 Debug: [ubuntu.media.mit.edu/ubuntu-releases/gutsy/ubuntu-7.10-server-amd64.iso] Copy to use: SERVER
11/10/07 22:46:37 192.168.1.6 Guest 0000000081 Debug: WWW Session sending server request in thread 214
11/10/07 22:46:37 192.168.1.6 Guest 0000000080 Debug: Server response contains 35 bytes of resource data
11/10/07 22:46:37 192.168.1.6 Guest 0000000080 Debug: WWW Session processing HTTP response in thread ec0 - response code 200
11/10/07 22:46:37 192.168.1.6 Guest 0000000080 Debug: Pragma: no-cache in server response - disabling caching
11/10/07 22:46:37 192.168.1.6 Guest 0000000080 Traffic 350 1172 1140 329 0s
11/10/07 22:46:37 192.168.1.6 Guest 0000000081 Debug: Server response contains 4140 bytes of resource data
11/10/07 22:46:37 192.168.1.6 Guest 0000000081 Debug: WWW Session processing HTTP response in thread 214 - response code 200
11/10/07 22:46:47 192.168.1.6 Guest 0000000080 Debug: Server closed connection in thread ec0
11/10/07 22:46:47 192.168.1.6 Guest 0000000080 Traffic 0 0 0 0 10s
11/10/07 22:46:47 192.168.1.6 Guest 0000000080 Terminated exit code 0

Thank you.
Randy

[adrien]

Hi Randy

The AV plugin will drip-feed the data, and also block range requests (since it can't scan a partial file).

It would probably pay to add a whitelist entry to KAV for this URL so it won't scan it.

Regards

Adrien

[Randy Baker]

I have created a whitelist entry to KAV for this site, with no success. I deregistered and uninstalled KAV as well, and the system was rebooted. Neither attempt has relieved this problem.

I have noticed that not only is this particular site dead slow if it ever does start a download, many sites continue to take considerable time downloading all images that form a page. Getting a video from ca.youtube.com or youtube.com is painfully slow.

If I shutdown WinGate and start up WinProxy, I do not have these problems. There is a significant and perceptible difference in performance between the two products on this system, even when WinGate is not performing AV checking. I have ensured that WinProxy has no latent processes still hanging around in the process table. The system has been restarted many times now with WinProxy disabled, and WinGate set to start as a service.

My WinGate Server is a Windows XP Pro system, fully patched. Hardware includes Intel 915 Dual Core processor, with 2Gb of RAM. My LAN NIC is 1GB, WAN side is 100Mbps. My link is cable, 8Mb download / 1Mb upload.

Other services running on this system include 2 low priority BOINC processes (not been a problem before), a DynamicDNS client, and an older version of WatchdogPC (8.4.1). There are a total of 33 processes running on the system with 560 MB of RAM in use. The processor is always running at 100% due to BOINC, but other than extra memory consumption for calculations, there seems to be no perceptible performance differences when BOINC is not running. Remote Desktop is enable to accept connections, and WinProxy 6, Acrobat Reader 7.0, VNC viewer, firefox, PuTTY and CoreFTP Lite are the only other installed applications, and are typically not running.

In Windows Security Center, Windows Firewall is off, automatic updates downloads, (I choose to install), and Virus Protection is not monitored. I do not have Windows Defender installed.

Any ideas?
Thank you.

[Randy Baker]

I forgot to add the following:

When I issue the command netstat -no, I see many connections, but none in the SYN_SEND state. I do see an ESTABLISHED status to 18.85..21.55:80 for the download from ubuntu.media.mit.edu, but it eventually disappears from the table, and the browser continues to believe it is waiting for a download.

I have also noted that the slow page loads issue does not necessarily take longer when the network is busy compared to when only one user is active on the network.

[Charles Silvia]

See my previous post "Download Problems" of 11/5. Your symptoms sound very simmilar.

If you de-license Kasperski and Puresight and retart the Wingate service does the situation improve??

If so, then you need to re-look at how you defined the download site in the allowed list of Kasperski and Puresight. When properly allowed the double storing of the files does not take place.

[Randy Baker]

Charles, thank you for your post as the issues I am facing are related to Kasperski and Puresight. However, I have over 65Gb of free disk space, so tricking data is the issue.

Since both of these plug-ins trickle data through, the perceived performance hit is unacceptable. As a WinProxy user for close to, if not 10 years now, I have been very happy with the performance of the Panda AV plug in, and the SmartFilter plugin.

I have always considered the Panda AV product as a different and extra AV layer at the perimeter of my network. All other windows systems have their own AV protection installed. Losing AV protection at the perimieter is not that big of deal.

However, site filtering is an important feature for my network, but I must also have the performance. My Ubuntu download was one ISO under 600Mb. Other ISO's I download are larger and may have 6 or 8 ISO's per set. The drip or trickle feature is too much of a hassle.

If I go with WinGate, I won't be going with Kasperski or Puresight.

[adrien]

Hi Randy

What browser client are you using? I'm wondering if you're hitting some other problem. We can try reproing this in our lab.

Sounds like it shouldn't be a CPU issue. We find most bottlenecks are network issues, and actually mainly site responsiveness.

Dripfeeding is handled by WinGate, rather than the plugins themselves (although this has changed for WinGate 7), so it shouldn't matter whether you are using both PureSight and Kaspersky AV or just one of them.

We've also found the performance of Kaspersky scanning to be very good. However, scanning a 500MB file (like that iso image) will take some memory.

Also do you by chance use NTLM auth in WinGate, and is the browser configured to connect to the proxy, or are you intercepting the connection in the proxy?

Regards

Adrien

[Randy Baker]

Here is an incomplete overview of the network.

Dell PowerConnect 2716
Gigabit Ethernet Switch, one VLAN configured
21 Days uptime, All error counters = 0

Windows XP Pro, Service Pack 2 (WinProxy Server)
- Pentium D 2.66 Ghz
- 2Gb RAM
- WinGate 6.2.2 (Build 1137)
- Internet Explorer 7
- LAN NIC 1 Gbps
- WAN NIC 100 Mbps
- ISP Rogers (cable) Extreme Speed (asymetrical), 8 Mbps download, 1 Mbps upload

Systems that had trouble downloading or very slow page builds when Kaspersky or PureSight installed. Issue encountered at multiple web sites, and multiple download sites. Systems did not have same issue when WinGate shutdown and WinProxy started.

Windows Vista Home Premium (Workstation 1)
- Pentium D 3.20 Ghz
- 2 Gb RAM
- Internet Explorer 7
- Firefox 2.0.0.9
- Opera 9.21
- 1 Gbps Ethernet

Windows XP Home, SP 2 (Workstation 2)
- Pentium D 3.20 Ghz
- 2 Gb RAM
- Internet Explorer 7
- Firefox 2.0.0.9
- 1 Gbps Ethernet

Windows XP Home, SP 2 (Workstation 3)
- Intel P4 2.4Ghz (hyperthreading enabled)
- 1Gb RAM
- Internet Explorer 7
- 1 Gbps Ethernet

Mac OSX (Leopard) (Workstation 4)
- Intel Core 2 Duo, 2.0 Ghz
- 2Gb RAM
- 1 Gbps Ethernet
- Safari 3

Red Hat Linux V5 (Server 1)
- Pentium D 2.66 Ghz
- 1 Gb RAM
- LAN 100 Mbps
- WAN 100 Mbps (disabled)
- Firefox 2.0.0.9

Hope this helps.
RB

[adrien]

I'm just wondering if the WinProxy and WinGate network drivers are conflicting somehow.

Even if you have WinProxy disabled, it's possible its driver is still loading.. Are you able to rename the WinProxy kernel driver and restart that machine and see if there's still a problem?

Regards

Adrien

[Randy Baker]

I renamed the WinProxy directory to ensure that nothing is loaded from there at boot time, and I checked Blue Coats website for information on drivers or file locations. I had no luck there.

I did reboot and I reinstalled PureSight followed by another reboot. Tried to start a download of an ubuntu ISO, but it looks like the whole file is going to be downloaded before I get prompted to run or save.

Just to prove that PureSight was working, I tried to hit playboy.com and penthouse.com, no problem, it was blocked by WinGate. I then went to google.com and searched for "sex" images with no filtering of results. That returned a lot of porn, and clicking on random sites allowed me through. I looked for some obvious and easy to type urls, and I manually typed in an URL, and I was still let through. I didn't try drilling further into the sites, but I should not have been able to get to the pages I did.

I checked the PureSight configuration, and guests are set for filtered, sexual content was checked and set to 100%. Settings appeared to be in order, and then I realized I assumed 100% meant no tolerance. So I set the slider to the default of 50% and retested. Now sites are blocked. Reset slider to 100%, now I get through. Set the slider back to 50% until I can get to do some additional testing as to what the % of certainty is satisfactory.

I would like to add my sentiment that I think the PureSight methodology needs to be reconsidered so that ISO images, as an example, don't appear to take forever to start downloading. One solution is to create an exception list based on file type. I know this isn't perfect, unless the file type is discerned from the file header and not the extension text. This might be more practical than creating exceptions by site since one download site might be too busy, but a mirror is fine. I also have my doubts that PureSight would be able to detect prohibited material within an ISO anyhow.

[adrien]

I renamed the WinProxy directory to ensure that nothing is loaded from there at boot time, and I checked Blue Coats website for information on drivers or file locations. I had no luck there.

Drivers typically live in the Windows\System32\Drivers folder. I've done a bit of googling but haven't found the name of the driver. You may unfortunately need to uninstall WinProxy to remove the driver.

I did reboot and I reinstalled PureSight followed by another reboot. Tried to start a download of an ubuntu ISO, but it looks like the whole file is going to be downloaded before I get prompted to run or save.

That will be Kaspersky AV. PureSight shouldn't be touching anything other than text/html content.

Just to prove that PureSight was working, I tried to hit playboy.com and penthouse.com, no problem, it was blocked by WinGate. I then went to google.com and searched for "sex" images with no filtering of results. That returned a lot of porn, and clicking on random sites allowed me through. I looked for some obvious and easy to type urls, and I manually typed in an URL, and I was still let through. I didn't try drilling further into the sites, but I should not have been able to get to the pages I did.

I checked the PureSight configuration, and guests are set for filtered, sexual content was checked and set to 100%. Settings appeared to be in order, and then I realized I assumed 100% meant no tolerance. So I set the slider to the default of 50% and retested. Now sites are blocked. Reset slider to 100%, now I get through. Set the slider back to 50% until I can get to do some additional testing as to what the % of certainty is satisfactory.

that percentage should be read as "how sure must I be that a page is porn before I block it", so if you set it to 100% it won't block anything except stuff it is 100% certain is porn (stuff on the pre-categorised list).

I would like to add my sentiment that I think the PureSight methodology needs to be reconsidered so that ISO images, as an example, don't appear to take forever to start downloading. One solution is to create an exception list based on file type. I know this isn't perfect, unless the file type is discerned from the file header and not the extension text. This might be more practical than creating exceptions by site since one download site might be too busy, but a mirror is fine. I also have my doubts that PureSight would be able to detect prohibited material within an ISO anyhow.

PureSight only scans text/html. However Kaspersky AV scans everything.

Adrien

[Randy Baker]

I have uninstalled WinProxy and rebooted. From past experience on a prior version, the WinProxy uninstall did not clean out all WinProxy Registry Entries. I don't know if they have resolved this so that all entries are deleted.

As for the curret state of WinGate, Kaspersky is not installed, but PureSight is. I am attempting to download an ISO, and my WinGate server disk is being filled up, and regretably, the download behaviour persists.

In reviewing the PureSight website, I found the following statement.

PureSight differs from products on the market today because it does not rely upon keywords or databases as its primary means of filtering. Instead, the primary filtering technology is based upon AI (Artificial Intelligence) which previews the content of a website, determines if it is inappropriate based on the engines experience and collected data, and blocks the site if necessary.

The behaviour I am seeing with WinGate seems reasonable that PureSight is attempting to preview this ISO as a site.

The next step may be to wipe the system clean and rebuild. That's not going to happen before the weekend though.

[adrien]

I wouldn't recommend a rebuild.

It should be possible to get that iso image to not be scanned by PureSight. if there is no Kaspersky AV in the mix, I'm struggling to see why PureSight would want to scan it, but is it possible the server serving the image is trying to send it as Content-Type: text/html ?

I'll have another look in the framework - this could be a bug, PureSight should shut itself off for that request.

[Randy Baker]

I attempted another download last night and then went to bed. This morning, the web page timed out, no dialog box to save or run the download.

The following is the URL for the download.

http://www.ubuntu.com/getubuntu/downloa ... t+Download

The following is the extract from the log files.

11/15/07 17:24:00 192.168.1.6 Guest 0000000672 Created:
11/15/07 17:24:00 192.168.1.6 Guest 0000000672 Requested: http://mirrors.gigenet.com/ubuntu/gutsy ... -amd64.iso
11/15/07 17:24:00 192.168.1.6 Guest 0000000672 Debug: [mirrors.gigenet.com/ubuntu/gutsy/ubuntu-7.10-desktop-amd64.iso] Copy to use: SERVER
11/15/07 17:24:00 192.168.1.6 Guest 0000000672 Debug: WWW Session sending server request in thread 794
11/15/07 17:24:00 192.168.1.6 Guest 0000000672 Debug: WWW Session processing HTTP response in thread 794 - response code 200
11/15/07 17:24:09 192.168.1.6 Guest 0000000669 Debug: Server closed connection in thread 374
11/15/07 17:24:09 192.168.1.6 Guest 0000000669 Traffic 0 0 0 0 10s
11/15/07 17:24:09 192.168.1.6 Guest 0000000669 Terminated exit code 0
11/15/07 17:24:09 192.168.1.6 Guest 0000000670 Debug: Server closed connection in thread 3ec
11/15/07 17:24:09 192.168.1.6 Guest 0000000670 Traffic 0 0 0 0 10s
11/15/07 17:24:09 192.168.1.6 Guest 0000000670 Terminated exit code 0

Is there anyway within WinGate to determine if the server is trying to send the ISO as Content-Type: text/html. If not, later tonight I can install tcpdump (microolap.com/products/network/tcpdump/) onto the wingate server to see what is coming down the pipe.

Randy

[Nev]

The following is the URL for the download.

http://www.ubuntu.com/getubuntu/downloa ... t+Download

Tried that URL in .au, all ok at the speed of my connection at least via Wingate Proxy // SeaMonkey // Win2k3.

[adrien]

The content type coming back from the server I get redirected to is

Content-Type: application/iso

So PureSight shouldn't be scanning it. I'll need to check the source code to see why if you have no KAV installed, WinGate still feels it necessary to spool the download to disk.

I'd expect also if you have caching enabled, for this to be put into WinGate's cache as well.

Adrien

[winproxy_guru]

For Compatability with Wingate's ENS or NAT Driver I recommend that you uninstall winproxy's DNE (Deterministic Network Enhancer) Which Sits on the nat stack as well. Its under your network card.