Virtual IP & Guest Redirects

I would like to shed some light on how exactly the guest redirect works when a guest user connects and is redirected to the WLC’s captive portal. This is one of those items that I always overlooked (because it was working) but was forced to learn when it was found the virtual IP address was causing problems. You can read about that here if you would like.

The following is the text from Cisco’s WLC Configuration Guide:

With Layer 3 security, the client will issue a DNS query for a web server; the controller spoofs the DNS
response, providing its own Virtual IP address. The client then opens a web connection (HTTP port 80 or
HTTPS port 443) to that address. The controller redirects(to itself [Local Web Authentication] or to an external
server [Central Web Authentication]), and then that dialog is used to authorize the client to connect to the
network.

Many times it is easier for me to understand something if it is laid out in a step-by-step, serial fashion. In order to help out the reader, I thought it would be best to capture some packets and illustrate the process of how a guest user is redirected to the captive portal.

**Packet Capture of Guest User Being Redirected to the Guest Captive Portal**

Upon connecting, the device uses DHCP to get an IP address. In the instance shown below, the device is requesting an IP address immediately instead of going through the full DORA process. Note the virtual IP is used as the source and in this case is being used as the DHCP relay (the WLC is the DHCP server).

The client’s OS is Windows 10 and Microsoft has a built-in method for testing whether or not the computer is connected to the internet when it detects a potential hotspot. It will send a packet to msftconnecttest.com. The guest wlan is configured such that 8.8.8.8 is utilized for DNS. Note the computer sends a DNS request for msftconnecttest.com and a response is given.

Now that the computer knows how to communicate to msftconnecttest.com, it next attempts to communicate with the server by sending an HTTP request to “4-c-0003-.c-msedge.net”. It is not shown below, but the response in packet 5 gave both the IP address as well as the URL. The packet capture tool used the URL in packet 6 instead of the IP address in the Destination column. The WLC spoofs the server in packet 8 and redirects the client to “guest.company.com”…very sneaky eh?

Packet 6 – **HTTP Request to msftconnecttest.com**

**Packet 8 – WLC Pretending to be msftconnecttest.com Redirecting to guest.company.com**

The client does a DNS lookup to be able to get to guest.company.com. This step is very important as an external DNS server needs to to have a DNS record pointing it to the virtual IP address. This caused some confusion because it didn’t make any sense that a DNS record was going to point to a virtual IP address that was not routable! Nonetheless, this is how the WLC and the client communicate…please note there is never any traffic on the distribution system addressed to this IP address. This traffic is only between the wireless clients and the WLC.

**DNS Lookup of guest.company.com – Response is Virtual IP Address of WLC**

Finally the guest device sends traffic to the WLC captive portal – guest.company.com (192.0.2.1) utilizing secure HTTP where the WLC can then authenticate the device and allow it access to the internet.

**Guest Client Communicating with Portal – guest.company.com at 192.0.2.1**

One final note on the HTTPS captive portal certificate. In order to avoid the certificate warning, the Web Auth Certificate needs to have the FQDN of the virtual IP address (exactly as the DNS record) listed as the CN. To make things a little easier, wildcard certificates are also accepted. In my case, “*.company.com” is used as the CN.

This article does not cover all the uses of the virtual IP address but hopefully it provides a little more understanding. Thanks for reading!

Virtual IP Woes!

I wanted to write about a problem I had recently with the virtual IP address of a Cisco WLC. The IP address had been in use for years with no issues. It was addressed in the 12.x.x.x subnet and was actually a real IP address for one of our external websites. I had read in the past the virtual IP address of a wireless controller should never be a real IP address as it may cause issues but it had been configured this way for years and it was left uncorrected. I was a little confused by how the virtual IP worked and thought why fix something that isn’t broke! 😉 This misconfiguration reared its head in a way I would have never guessed.

Before I can jump in and give you the problem and solution, I need to provide some background. In addition to managing the WiFi network, I have the responsibility of managing our fleet of radios across the U.S. There are thousands of radios that need to be managed and maintained. Keeping inventory of these radios and ensuring these devices are in compliance can be a real challenge! Fortunately with the Motorola Solutions Mototrbo line of radios, managing these is greatly simplified using a radio management server and WiFi. The radios automatically connect to their SSID and the radios can be read from or written to as long as they are connected to the wireless network…all from a single pane of glass.

Each of the Mototrbo radios have a unique radio ID along with a CAI (Common Air Interface) Network and a CAI Group Network value. While they operate in digital mode, the radios communicate with other radios by one of two ways: a group call (talkgroup) or a private call (one-to-one). When the radios make a private call, the radio combines the radio ID with the CAI value, which is defaulted to 12. When combined, this creates an IP address. For example, radio ID 1 combined with its default CAI of 12 will be 12.0.0.1. You can actually ping radios from your computer! Please note this is not the WiFi address of the radio; it is the radio’s CAI or the main radio interface used to communicate with other radios over VHF, UHF, or 800/900 MHz. The radio has a WiFi interface, a USB interface, and its main radio interface (which is the CAI). Traffic coming into the radio via its WiFi or USB interface that is destined for a 12.x.x.x network will be routed out the CAI. Group calls are routed the same way but the difference is that Group calls default to a 225.x.x.x IP address.

Ok, now that you have that background info I can tell you the rest of the story. One day I received a call stating the Mototrbo radios were not operating correctly. They were very intermittent. Upon investigation it was noted the repeaters seemed to be receiving and retransmitting intermittently and sometimes it was quite often. The random transmissions would come and go with a high enough duty cycle preventing all valid two-way handheld transmissions. It was a big problem. The radio dealer was called out and they investigated and reported back they thought the problem arised when the radios were connected to the WiFi. Honestly I thought it was a bug in the radios but by carefully observing the radios when they connected it was found that when the radios received traffic from the virtual IP address of the WLC (think DHCP on the WLC), the radios simply tried responding to the DHCP server but instead routed the response out its CAI! The result was a bunch of digital traffic being burst out and being repeated on the repeaters taking up airtime.

**Radio Routing Traffic Destined to 12.x.x.x Out the CAI**

Once it was understood what was happening, the fix to all of this was to change the virtual IP address on the WLC. We changed it to 192.0.1.2 and the problem went away. I will post another article on the virtual IP and how it is used with the captive portal. This topic always confused me a bit so I hope I can help somebody out there that is still fuzzy on this topic. Thanks for reading!

Slow WiFi

One day I received an email stating one of our warehouses had very slow WiFi. As a matter of fact, the WiFi was never any good. After following up with the customer, I decided to go to Prime Infrastructure and view any historical data from the AP3702s that were installed in the warehouse. Right away I could tell there were some issues as the channel utilization was very high in the 2.4 GHz band. Shown below is an example of what I was seeing.

I know from experience that channel utilization in the 2.4 band is normally higher than in the 5 GHz space. Normal channel utilization will usually be anywhere from 10 to 35% as this spectrum tends to be noisier just due to the presence of other competing devices, especially in the office areas however this was a warehouse and they tend to have a quieter environment. Utilization as high as 65 – 70% usually means WiFi devices are either not talking or data throughput is very slow and requires some correction to get this number back down.

I decided to put one of the access points in SE-Connect mode. In this mode a separate application called Cisco Spectrum Expert can be run on your PC. This application connects to the access point and the spectrum data from the CleanAir-enabled access point can be streamed in real-time from the access point to your PC. It is like having a spectrum analyzer at the site and works pretty well.

Once the AP was in SE-Connect mode (a reboot is necessary), I copied the IP address and the Network Spectrum Interface key. This information was easily found by going to the WLC GUI and selecting the access point in the All APs>Details page. This information is needed for the Spectrum Expert application and is what is utilized to make the connection to the AP. The radio (a or b) will need to be selected upon connecting as well. Once connected you can choose to display different FFT or spectrogram plots. I chose to look at a combination of FFT Duty Cycle, Real-Time FFT, and a couple other plots to understand the percentage of time the RF medium was being utilized over time. Shown below is what I seen that day.

The numbers I seen from this access point were pretty shocking in that it was reporting 100% duty cycle for long periods of time. You can see from the plot on the lower right that the duty cycle was at or near 100% for over 10 minutes! You can also see from the plots on the left this interference is utilizing the entire 2.4 GHz band and is not your typical microwave oven incident. After calling the customer back and asking them about any other wireless devices they had in the warehouse (they did not know of any) I decided an onsite visit was in demand.

Upon arrival I fired up my PC and ran AirMagnet’s Spectrum XT. This is a great application. It utilizes a USB dongle that houses a spectrum analyzer on a chip. It is easy to use and you can record any data you receive to easily play back later or for documentation purposes. Right away I was faced with the similar high duty-cycle and interference I seen from the access point and started confidently walking the warehouse. I remember thinking I will have this interference tracked down in no time! I was quickly humbled because the interference would mysteriously come and go leaving me somewhat baffled as to where it was coming from. I remember going back in forth in the warehouse and going towards what I thought was the source and then once I thought I was on top of it (the signal levels would get as high as in the -40s!), it would be decreased or even gone. After doing this for some time I felt defeated and decided to capture some packets to get another perspective.

I then opened up Savvius Omnipeek and captured some packets in the 2.4 GHz band. Right away I spotted multiple access points from another neighbor that was still utilizing old 802.11b rates. This explained some of the utilization but not all. The duty cycle was too high for too long and there were not that many clients in the area. I decided I needed to go back to the spectrum analyzer and figure out the source of the problem.

Once again I started walking around the warehouse until I came to a spot that was in the high -30s! I knew I was close. I spun around in all directions only to notice I was near a forklift. All of a sudden it hit me…there must be something on the forklift causing this interference. I immediately started to examine the forklift to determine what wireless equipment they were using. I spotted a forklift mounted computer and noted the barcode scanner but knew this was not the source. I continued to inspect the equipment and noticed a mysterious box that had a little antenna connected to it. I disconnected the power to determine if the noise went away and it did. Finally I found the culprit! Here is what I found:

I then performed some Google detective work and found the manufacture of the device. There are two devices installed on the forklift: an video receiver and a transmitter. A camera is mounted on the forks and the output is sent to a video transmitter that is also mounted on the forks. The transmitter is linked to a receiver that is mounted down by the operator. This video is then sent to a monitor that is used by the operator to tell exactly where his forks are at all times.

I exchanged some emails and it was determined neither the cameras nor the neighbors 802.11b access points were going to change. The forklift clients were only 2.4 GHz capable and were due for an upgrade. The best thing to do at this time was to replace the clients with dual-band units that could utilize the 5 GHz. After the clients were replaced, I shut down the 2.4 GHz radios and the barcode scanners immediately started using the 5 GHz radios. The operators stated they never really had good WiFi communications before this so they were pretty excited!

Here is the data sheet for the video transmitter/receiver:

Transmitter
Working Voltage	DC 10-32V
Video System	AUTO
Working Frequency	2400-2483.5MHz
Working Distance	120M
Video Decode	MPEG4
Transmitter Sensitivity	:-89dBm
RF Bit Rate	4Mbps
Spread Spectrum	FHSS
Time Delay	120mS
Waterproof Rate	IP66
Working Temperature	:-20~+70C RH90%
Storage Temperature	:-30~+80C RH90%
Receiver
Working Voltage	DC 10-32V
Video System	AUTO
Working Frequency	2400-2483.5MHz
Working Distance	120M
Video Decode	MPEG4
Receiver Sensitivity	:-17dBm
Bit Rate	4Mbps
Spread Spectrum	FHSS
Time Delay	120mS
Waterproof Rate	IP66
Working Temperature	:-20~+70C RH90%
Storage Temperature	:-30~+80C RH90%

If these types of units are going to be used, I would recommend something that is not going to be used in either WiFi band (or at least be WiFi friendly) and verify they do indeed do what they are advertised to do.

After looking back at this, I thought I should have caught this sooner. It just didn’t dawn on me the source of the noise would be installed on the forklifts. This was not standard equipment and there are better solutions that should have been used. Please let me know if you have ever ran into this issue.

IT/OT Convergence and WiFi

Today I thought I would share some ideas around IT/OT Convergence and WiFi. IT/OT Convergence involves the idea of sharing common network resources in an effort to decrease the management burden and costs normally associated with operating two separate networks while maintaining the same level of security (or keeping it as high as possible). IT and OT networks have been historically isolated and segmented from each other to ensure the compromise or failure of one has no impact on the other. Having separation typically means duplicate infrastructure in each segment and also requires two different groups of people to maintain them.

WiFi and mobile computing has really opened up all sorts of possibilities in the industrial area. One use case in high demand is the mobile worker that can wirelessly monitor and control processes directly over his mobile laptop or tablet. No longer does a fixed workstation have to be manned at all times. The wireless remote operator frees up this individual and allows this person to go mobile and perform checks and maintenance throughout the plant or factory floor while still having the ability to control and monitor everything going on in his process network. One man can now do the job of what used to require two.

This brings me to my next point – Security. Obviously the implementation of WiFi on the plant floor is a requirement for the wireless mobile worker. WiFi utilizes an unbounded medium and introduces many possible vulnerabilities and increases the attack vector many times that of the traditional fixed wired workstation in a control room. If the WiFi network is setup without much thought, not only will the OT network be less secure and prone to attack but the management burden can be increased.

If the OT network is segmented from the IT network (as is usually the case) and somebody wants to implement WiFi, the first thought is to perform a survey to figure out where the access points should go and throw in a WLC. Alternatively you could install access points without a centralized WLC and utilize something like Cisco’s Mobility Express. The problem with having a WLC at each local OT network is there needs to be somebody who can manage each one, thus increasing the management burden. Management of a wireless network is often overlooked and not thought of until a vulnerability comes out (KRACK) and all of a sudden comes the realization that you have to upgrade twenty-five different WLCs. This does not scale well unless you have a team of wireless engineers but when is this the case? You can use Mobility Express but this solution still has limited options. For instance, you still need to maintain the individual sites configuration and there are no anchoring capabilities. Following the NIST’s standards of keeping traffic segmented properly, if it is desired to propagate an IT SSID in the same OT environment, a second AP that is connected to an IT network will be required. The ideal is to install a WLC in a shared process network. With a shared WLC, the management burden is reduced (one WLC upgrade!) and IT SSIDs can be anchored to an IT WLC. For the case of IT Wlans using 802.1X, the AAA can even be proxied or relayed to an IT radius server keeping both the management and data traffic segmented. All of that is good but if you do not have that shared OT network, you may be in for a bit of a struggle as the fight is real when it comes to the IT/OT network segmentation debate!

I stated the previous as a backdrop to illustrate an alternative solution that truly does utilize the IT/OT convergence principles and overcomes some of the burden. The answer is to utilize a combination of the local FW that segments the networks, Cisco AnyConnect, and the local OT domain controller. Admittingly this solution takes a bit of work however if you need to provide a secure solution and one that will work whether or not the WAN is down, then I propose the following as one solution.

As already stated, a requirement of this solution is to ensure the Wlan is available at all times, whether or not the AP can reach the centralized shared WLC or not. For this reason, the only wireless security policy that will work is WPA2 or WPA3 (PSK). FlexConnect is also required to ensure if the WAN does go down and the WLC becomes unreachable, the Process clients will still function normally.

The firewall is configured to have a dedicated trunked wireless interface consisting of an AP management Vlan and a Client Vlan. Firewall rules are implemented such that all traffic is blocked except CAPWAP traffic going to the IT WLC (UDP Port 5246 & 5247) on the AP management interface. A deny any any is applied on the Client interface blocking all traffic.

Remote Access is configured for the client to be able to VPN into the Process network via Cisco AnyConnect. The client IP address is translated via NAT and ACLs are put in place to only allow the client specific destinations. This can be done utilizing Dynamic Access Policies. Both the ACL and the appropriate AD group can be added here to properly permit or deny access based on the AD group membership. Finally, an AAA server group is added with LDAP utilized to query the local Process domain controllers.

On the IT WLC, a Process SSID is configured as locally-switched with a PSK. Client data is offloaded at the AP in a locally-switched Wlan. The access points are configured for FlexConnect and the appropriate Wlan is mapped to the corresponding Vlans. If IT SSIDs are to be propagated on the AP, these will need to be configured as centralized Wlans so that the client traffic will be tunneled up to the IT WLC.

The next step to this process is to configure the client device (Windows). Cisco AnyConnect is installed on the device and then configured to automatically connect to the Process SSID. Once associated, AnyConnect is used to VPN into the local Process domain utilizing their user credentials. This ensures that even if somebody knows the PSK, they will still need Cisco AnyConnect installed and be able to connect using somebody’s domain credentials. If security still seems to be too low, certificates can be utilized as a secondary form of authentication (Multi-Factor Authentication). If the device is lost or stolen, the user account or the device certificate can be revoked to prevent the device from connecting to the Process network. Here are the steps to connecting to the VPN:

Let me know what you think!