Sometimes even a technology company can develop technology based challenges right in its own office that defies the logic of a whole lot of deeply knowledgeable industry professionals. In this case, the life blood of our daily work; our internet bandwidth; seemingly dried up.

Setting the Stage

BridgePoint leverages a fiber network that powers our communications, with a standard average speed about 100 megabits, we rarely have complaints about access here in the office. While most of our server network infrastructure is based out of a secure data center a few miles away or in Microsoft Azure, we still host many servers for redundancy and testing so anything that impacts our ability to get online, communicate with our data centers or customers, grows from annoyance to a priority very quickly.

Over the past week, more and more complaints were coming in about very slow loading websites. At first it seemed random and inconsistent which as IT professionals know, causes hair pulling because if you can’t reproduce it consistently, it makes tracking down the issue that much harder.

We Hit Rock Bottom

Yesterday was that day. Tuesday before Thanksgiving and the phone is ringing with clients trying to clear items, employees trying to get business done prior to taking much earned time off and essential downtime planning for clients who need to leverage the holiday to implement new services. Our lead network guru Fred Barzycki had been slowly narrowing the possible points of failure in the previous few days but now it was serious. Fred dropped almost everything he had going on and began in earnest to pull apart vital network parts and pieces in hopes of finding an answer. Network traffic monitors showed no high traffic issues (chatty programs, virus, malware, downloads/uploads), the fiber data provider had been checked and ruled out any connection issues outside our network and non essential devices had been turned off already. No change, our once 100 megabit connection read as 700k download to some of our users!

Firewall Admin

After nearly a full day of changing switches, checking and restoring configurations and tediously going through screens of settings and resets Fred notes that for HIS machine, he is suddenly working at full speed. Problem is, nobody else is and yet nothing seemingly had been changed. Turns out the FIX had something to do with being logged in because as soon as he logged out of the firewalls web based administration, speeds ran right back down to a crawl. Repeat tests confirmed that simply logging into the Dell SonicWall administration site suddenly cleared up the problem as long as you were logged into the admin. Once you log out, back to slow speeds.

The questions plaguing the team now was WHY and HOW? We had found the source of the issue, could replicate the issue and the fix on any machine in the network. The annoyance was that there seemed to be nothing in the settings that would account for user slowdown. Logically Content Filtering is the first suspect on a firewall especially when an Admin can get full speed and others cannot except our Content Filtering wasn’t being used (we’re all professionals and no reason to restrict usage btw).

Sometimes the best solution is Google

Now that we had the issue focused down to one point that we could replicate, a few hours of pouring over settings was getting no results. Armed with the details we DID have, we turned to the Internet to see if our problem (and our solution) might lurk out there. Turns out, we were not alone.

The Solution

After some digging a community post from February of 2014 pops up “sonicwall seems to slow down network when I’m not logged in

I have a client who has 150 Mb down speeds from the cable company (I know, here we go).

From the modem it runs via speedtest.net as advertized. However, on the LAN side of the SonicWall, it run at around 75 Mb, and that’s fine.

The problem, is that once a week, or sometimes even a few days a week, the speeds drop way down to below 20 Mb, but if I log into the SonicWall, they immediately pick back up (then drop again once my login times out or I log out).

I can’t figure it out to save my life. Any and all advice would be super helpful.

Some Particulars:

If I go to speedtest, the initial ping will hang. If I do nothing the test times out entirely. If I immediately log into the SonicWall, the test suddenly completes it’s latency test (ping) and then works as intended.

Speedtest.net’s ping times range from 16 ms to 22 ms.

The SonicWall is a TZ 215.

I do not see any errors in the logs.

On the External side, I have set the Link Speed to Auto Negotiate.

On the Internal Side, I have set the Link Speed to 1 GBps, FULL Duplex. (Both show a status of 1 Gbps Full Duplex).

Contend Filtering via the Content Filter Service is on.

Security Services Settings is set to Performance Optimized.

Gateway AV and Anti Spyware are both OFF.

Gateway IPS is ON.

CPUs run about 8 to 12 %, connections peaked at 479. Connection usage is 0.253%.

Firmware is SonicOS Enhanced 5.8.1.13-1o

Thank you everyone for any advice.

The user ended up finding his own answer and it was there we got our help

After contacting support, trial and error and communication failures with the client, I finally figured this out.

I knew early on that it was a high level issue – pings did not drop or change latency during these “outages”. I happened to be on site and logged into the SonicWall and noticed the issue immediately cleared up on that workstation – but not any of the others.

So, because I’m an IT guy, I logged everyone (there’s only 8 users) into a non configuration session and minimized the screen and that seems to solve the issue completely… except that I would have to do this everyday.

That helped narrow my search – what would require ALL users to log into a SonicWall session in order to access the internet…

Content Filtering.

Yep, someone who came before me had set up the SonicWall to do content filtering via screens – which basically means you have to log into a SonicWall access screen to access the internet. However, none of it was set up, they just turned it on and walked away. I guess now we know why he’s the previous admin and I’m the new one eh?

Anyway, switching it over to app mode cleared it all up completely.

Off To Test

Logging into our SonicWall admin we once again confirmed that no content filters were explicitly set for users but YES, our  setting matched that mentioned and so we made the switch and hit save.

Logging out and user testing confirmed speed tests returned fast full bandwidth across the board. The issue is this ONE setting which we are more than happy to share in case others ever develop a similar issue. Simply set the CFS Policy Assignment to “Via App Rules”

SonicWall-ViaAppRules

In Conclusion

Roughly two weeks ago a firmware update was applied to the SonicWall, we suspect that in that update the Content Filter setting may have been set to a ‘default’ or another base setting was reset that created the issue. Now that we know what to look for, we’ll keep our eye on it. BridgePoint knows full well that when essential communications are down for our clients, it means lost revenue and increased frustration. While we’re never happy to have it happen (especially to our own network) it is great to track and solve a complex issue and gain the confidence that we can continue to do the same for our trusted client partners.