Slow WiFi

One day I received an email stating one of our warehouses had very slow WiFi. As a matter of fact, the WiFi was never any good. After following up with the customer, I decided to go to Prime Infrastructure and view any historical data from the AP3702s that were installed in the warehouse. Right away I could tell there were some issues as the channel utilization was very high in the 2.4 GHz band. Shown below is an example of what I was seeing.

I know from experience that channel utilization in the 2.4 band is normally higher than in the 5 GHz space. Normal channel utilization will usually be anywhere from 10 to 35% as this spectrum tends to be noisier just due to the presence of other competing devices, especially in the office areas however this was a warehouse and they tend to have a quieter environment. Utilization as high as 65 – 70% usually means WiFi devices are either not talking or data throughput is very slow and requires some correction to get this number back down.

I decided to put one of the access points in SE-Connect mode. In this mode a separate application called Cisco Spectrum Expert can be run on your PC. This application connects to the access point and the spectrum data from the CleanAir-enabled access point can be streamed in real-time from the access point to your PC. It is like having a spectrum analyzer at the site and works pretty well.

Once the AP was in SE-Connect mode (a reboot is necessary), I copied the IP address and the Network Spectrum Interface key. This information was easily found by going to the WLC GUI and selecting the access point in the All APs>Details page. This information is needed for the Spectrum Expert application and is what is utilized to make the connection to the AP. The radio (a or b) will need to be selected upon connecting as well. Once connected you can choose to display different FFT or spectrogram plots. I chose to look at a combination of FFT Duty Cycle, Real-Time FFT, and a couple other plots to understand the percentage of time the RF medium was being utilized over time. Shown below is what I seen that day.

The numbers I seen from this access point were pretty shocking in that it was reporting 100% duty cycle for long periods of time. You can see from the plot on the lower right that the duty cycle was at or near 100% for over 10 minutes! You can also see from the plots on the left this interference is utilizing the entire 2.4 GHz band and is not your typical microwave oven incident. After calling the customer back and asking them about any other wireless devices they had in the warehouse (they did not know of any) I decided an onsite visit was in demand.

Upon arrival I fired up my PC and ran AirMagnet’s Spectrum XT. This is a great application. It utilizes a USB dongle that houses a spectrum analyzer on a chip. It is easy to use and you can record any data you receive to easily play back later or for documentation purposes. Right away I was faced with the similar high duty-cycle and interference I seen from the access point and started confidently walking the warehouse. I remember thinking I will have this interference tracked down in no time! I was quickly humbled because the interference would mysteriously come and go leaving me somewhat baffled as to where it was coming from. I remember going back in forth in the warehouse and going towards what I thought was the source and then once I thought I was on top of it (the signal levels would get as high as in the -40s!), it would be decreased or even gone. After doing this for some time I felt defeated and decided to capture some packets to get another perspective.

I then opened up Savvius Omnipeek and captured some packets in the 2.4 GHz band. Right away I spotted multiple access points from another neighbor that was still utilizing old 802.11b rates. This explained some of the utilization but not all. The duty cycle was too high for too long and there were not that many clients in the area. I decided I needed to go back to the spectrum analyzer and figure out the source of the problem.

Once again I started walking around the warehouse until I came to a spot that was in the high -30s! I knew I was close. I spun around in all directions only to notice I was near a forklift. All of a sudden it hit me…there must be something on the forklift causing this interference. I immediately started to examine the forklift to determine what wireless equipment they were using. I spotted a forklift mounted computer and noted the barcode scanner but knew this was not the source. I continued to inspect the equipment and noticed a mysterious box that had a little antenna connected to it. I disconnected the power to determine if the noise went away and it did. Finally I found the culprit! Here is what I found:

I then performed some Google detective work and found the manufacture of the device. There are two devices installed on the forklift: an video receiver and a transmitter. A camera is mounted on the forks and the output is sent to a video transmitter that is also mounted on the forks. The transmitter is linked to a receiver that is mounted down by the operator. This video is then sent to a monitor that is used by the operator to tell exactly where his forks are at all times.

I exchanged some emails and it was determined neither the cameras nor the neighbors 802.11b access points were going to change. The forklift clients were only 2.4 GHz capable and were due for an upgrade. The best thing to do at this time was to replace the clients with dual-band units that could utilize the 5 GHz. After the clients were replaced, I shut down the 2.4 GHz radios and the barcode scanners immediately started using the 5 GHz radios. The operators stated they never really had good WiFi communications before this so they were pretty excited!

Here is the data sheet for the video transmitter/receiver:

Transmitter
Working Voltage	DC 10-32V
Video System	AUTO
Working Frequency	2400-2483.5MHz
Working Distance	120M
Video Decode	MPEG4
Transmitter Sensitivity	:-89dBm
RF Bit Rate	4Mbps
Spread Spectrum	FHSS
Time Delay	120mS
Waterproof Rate	IP66
Working Temperature	:-20~+70C  RH90%
Storage Temperature	:-30~+80C  RH90%
Receiver
Working Voltage	DC 10-32V
Video System	AUTO
Working Frequency	2400-2483.5MHz
Working Distance	120M
Video Decode	MPEG4
Receiver Sensitivity	:-17dBm
Bit Rate	4Mbps
Spread Spectrum	FHSS
Time Delay	120mS
Waterproof Rate	IP66
Working Temperature	:-20~+70C  RH90%
Storage Temperature	:-30~+80C  RH90%

If these types of units are going to be used, I would recommend something that is not going to be used in either WiFi band (or at least be WiFi friendly) and verify they do indeed do what they are advertised to do.

After looking back at this, I thought I should have caught this sooner. It just didn’t dawn on me the source of the noise would be installed on the forklifts. This was not standard equipment and there are better solutions that should have been used. Please let me know if you have ever ran into this issue.