ArubaOS In-Service Upgrade with Clusters

It’s been a while between posts <insert contrite apology and excuses>. Hopefully the length and breadth of this one will make up for the lag.

How many of you out there have had challenges getting a maintenance window to perform a software upgrade on your wireless infrastructure?  We all know that the wireless network is becoming as mission-critical as the wired one, and that means getting an outage to perform preventative maintenance like a software update is becoming even more difficult.

Enter the ArubaOS 8 feature: In-Service Upgrades. Using the Cluster feature introduced in ArubaOS software version 8, the resiliency and redundancy that a controller cluster offers also grants the ability to perform software updates to the infrastructure without any outage or service impact.

Quick primer on Clustering:

Two or more Mobility Controllers (now referred to as Managed Devices in AOS8 architecture) can be formed into clusters to provide network resiliency; up to 12 MDs in a cluster.  Access points and client devices now form tunnels to both a primary and a standby MD, and should the primary MD become unreachable, fail over to the standby, pre-established tunnel to another MD in the cluster.  This topology allows for the obvious network resiliency and hitless client failover, but also seamless campus roaming, client load balancing, and In-Service Upgrades.

Here’s why this feature can be so important, besides the convenience of no more CAB meetings begging for a maintenance window at 2am:  this week, on Monday, we had the public revelation of the WPA2 vulnerability known as KRACK (for reference: https://www.krackattacks.com/ ).  Every vendor, both infrastructure and client devices, has been working behind the scenes for weeks and months to craft a software patch to mitigate this particular threat.  With an AOS8 cluster, Monday morning, while sipping your first coffee, you could have downloaded and updated all your WLAN infrastructure with the patch Aruba had available last week that mitigates the Krack infrastructure risk.  So instead of begging for an emergency outage from your boss that night, you could have said “no worries boss, got it covered, we’re good, already patched everything.”

All right, now we’re level-set on what clustering does, why it’s good and how it facilitates in-service upgrades, here’s the meat and potatoes of the procedure.  I ran through the full upgrade in my home lab to catch screenshots to explain.

Environment:

  • Software – current 8.1.0.3, going to 8.1.0.4 (Krack-proof)
  • Single Mobility Master (the admin and orchestration part of the network) – running in VM
  • One 7010 and one 7008 Mobility Controller, configured to be in a cluster.
  • Various models of APs, the majority running in Local mode for clients
  • Several clients connected to the “corporate” SSID

First step is to get that Mobility Master (MM) upgraded, which is a straightforward process.  The MM doesn’t terminate APs or clients, so it can really be taken out of service at any time and upgraded without any end-user impact – be as rough as you want.

PreMM upgrade

A simple file transfer, reboot of the partition holding the new code, and voila, a MM running the new code:

PostMM upgrade

With that part done, we now focus on the Managed Devices, the controllers, and APs.  AOS8 now has a hierarchical structure for managing all the nodes, so for this process, you’ll want to initiate the cluster upgrade from the Managed Network folder:

 PreMDupgrade 1

Pick your cluster:

PreMDupgrade 2

Define the FTP server info and the software version – **note the syntax of the Upgrade to version field**:

 PreMDupgrade 3

Last step, define the partition to write the new code to.  Aruba controllers have two boot partitions, so you can load this to the one that is not currently in use, giving you a back-out:

PreMDupgrade 4

And there you go.  That’s literally all you do.  Once you click the little blue button “Upgrade Cluster,” there’s no operator interaction, everything is orchestrated and automated.

But rather than just take my word for it, here’s some screenshots and narrative:

MDupgrade 1

The APs need to be balanced across the MDs so that one MD can have the new software downloaded, applied and reboot, without taking any clients or APs down. Once that’s done, in this case all the APs and clients shifted to the 7008, the 7010 started downloading the 8.1.0.4 code from my FTP server:

MDupgrade 2

Which completed successfully, and so the 7008 MC got its download of the code to the inactive boot partition:

MDupgrade 3

With both MDs getting the new software downloaded, the one without any APs or clients, the 7010, now reboots back into service with the partition containing the new code loaded:

MDupgrade 4

While that’s occurring, you can see the active MD, the 7008, has all the APs and clients terminating to it:

MDupgrade 5

Once the 7010 rebooted, it’s now running the new code, so it’s time to move all the APs and clients over to it so the 7008 can also get the upgrade.  But those APs first have to be upgraded too!  Here’s where an Aruba feature, ClientMatch, helps. The system uses this feature to find a best “alternate” AP for a client to move to, and the client is encouraged (read forced) to roam to another AP without dropping their connection.  Once an AP is unloaded of clients, it reboots using the preloaded 8.1.0.4 image:

MDupgrade 6

MDupgrade 7

Now that the 7010 controller and all the APs are all upgraded to the new code, the 7008 will reboot back into service using the 8.1.04 code that was previously downloaded:

MDupgrade 10

Abracadabra, that’s it.  We can see from the Mobility Master CLI the status of the cluster upgrade “Successfully completed.”  The same message in the GUI, the MM, the MDs, and all the APs are upgraded to the latest code.  All whilst not dropping a single client, or interrupting their connection:

MDupgrade 11

MDupgrade 12

After high-fiving myself for a job well done, and the applause dies down, it’s worth noting that the whole process does take some time to fully complete; this small environment took 37 minutes.  From my perspective, it could take 8 hours and I wouldn’t really care if it meant I patched my infrastructure without taking it out of service.

Wouldn’t you know it though, Aruba just dropped software version 8.2 with some really cool new features I could sure use.  Guess I’ll have to go to CAB and get a maintenance window.  Wait a minute….

**Post script: If you want to see this in-service upgrade in action on a much larger scale (2000 or so clients), check out the video in this link below from the Aruba Atmosphere conference in Nashville this past spring (best part starts around the 12 minute mark).

http://www.arubanetworks.com/video/?v=/atm17/Atm17AMER-TechDemo-ArubaOS8_The_foundation_for_the_next_decade.mp4&width=960&height=540&t=Atmosphere%202017%20ArubaOS%208%20Demo:%20The%20foundation%20for%20the%20next%20decade

 

 

 

 

 

 

 

Advertisements

Fun With Infrastructure Spectrum Analyzers

…or how I’m spending my Easter Weekend

I’ve gone dark the past few weeks, or a month plus on the blogging, the Twitter, the Slack, the gym … ok maybe not the gym.  Reason (not an excuse) is that I recently changed jobs and employers, and the change has forced my entire focus into the new gig (in a real positive way). Slacking off on all the other parts has been a bit of a drag. However, a wise man (@jockowillink) would say: new job taking time away from all the social media and the Interwebs; Good, it’ll create opportunities and be fodder for things to write about.  So here we are, me full of fodder.

One of the perks of working for a manufacturer like Aruba, is the crates of lab gear that rolls on up to my door to build a home SE lab.  I know my UPS driver, Rick, quite well after a month of near-daily visits.  Setting up the catalogue’s-worth of gear is/has been a lot of fun, not to mention a great refresher and learning experience.  I’ve always had a home lab, but now I have a Cadillac-grade lab (until my power bill comes in).  Having a lab for so long, like most can relate, I have some legacy and older bits lying around.  So when I fired up my shiny new Aruba Mobility Controllers I found myself connecting not only the latest and greatest Access Points, but for the heck of it, some older APs I had on the shelf.  Thing is, a decade-old AP-105 can’t match a new AP-335 for performance, so what’s the point of having these old APs in service? Ahhh, why not do what you can’t often do at a customer site, convert them to full time Spectrum Monitors and see what I can see?  The reason you don’t see APs deployed in a dedicated, non-client-serving mode typically is because customers are always budget conscious, and you get odd looks and head-shakes when you tell them you’d like to install APs that sit there and don’t serve clients. I know you can configure Aruba APs in Hybrid mode so they scan the client serving channel and serve clients, but I’m talking full-time, all-channel spectrum analysis.

Now why would this be valuable?  When it comes to spectrum analysis – the Layer 1 (Physical) monitoring of the RF spectrum – there’s really two different ways to get it done.  The first, and most common, is the “laptop” method, which means installing a special piece of software like Metageek’s awesome Chanalyzer, plug in a special WiFi adapter, and go walk around the area you want to scan.  This works great when both the tool and the resource who has the specialized­ knowledge to operate and interpret results are located at the site you want to scan the spectrum.  What happens when you have a distributed wireless network spread out over the entire province, or country, or world?  That’s a lot of frequent-flyer miles for the laptop and the person using it.  Not to mention, when you walk around with the laptop, you’re really only getting a “snapshot in time” view.  What happens if the interference is intermittent, like the knocking under the hood of your car? Here’s where the value of infrastructure spectrum analysis comes in.

Here’s how I did it, and it’s actually really easy.  Aruba Mobility Controllers operate in a lot of ways in a hierarchical manner.  Once you get how everything connects up to the next layer, it’s pretty easy.  A building block for the APs is the AP group.  APs get assigned to an AP Group, which in turn points to and defines various parameters like the SSIDs it will serve, and common RF parameters.  It’s under RF Management that you change on a per-radio basis the mode which the AP(s) in the group will operate.  One trick that you have to watch for is to change and define a new Radio Profile to nest inside the AP Group, or you’ll end up modifying the default, and that’ll likely mess things up in other Groups:

AP Group

And there you go, the MC will tell you it’s going to Apply and Reboot the APs, and once they come back up, they’re full-time Spectrum Analyzers.  You can also convert an individual AP for a one-off by going into the AP’s spectrum override profile if you’re not into the whole AP Group thing.

Now that you’ve converted one or more APs into Spectrum Mode, it’s time to observe the data.  Clicking the main Monitoring tab, and scrolling down to Spectrum Analysis, you get a new browser window pop up.  When you first bring the Spectrum Dashboard tab up, there’s nothing running and when you try to view anything it tells you that no spectrum monitors are assigned.  Have to fix that first under, surprisingly, the Spectrum Monitors tab, where you add the individual radios contained in the AP(s) that you assigned to the Spectrum Analysis AP Group:

SpecAn Add

Now we have a live AP, in Spectrum Analysis-mode, and it’s assigned to the Spectrum Analysis Dashboard – DISCO!  Let’s see what it sees.  Under the Spectrum Dashboard you’ll get a grid of four dashlets, and you have three different views in this format.  Now here’s what I really like; you can completely personalize what you see, and the source.  In the examples below I did the following: left-hand side was the 5 GHz radio, and the right-hand side was the 2.4 GHz radio.

  • View 1: Real-time FFT & Swept Spectrogram
  • View 2: Channel Metrics & Channel Summary Table
  • View 3: Active Devices & Active Devices Table

View 1

View 1

View 2

View 2

View 3

View 3

All in all, there are fourteen different dashlet options that you can visualize various data views.  Want to record the data over lunch time while you go grab a sandwich, salad and protein shake, just click Record, and when you come back, flip over to Playback View to see all that RF action you missed through your own Spectrum Analysis VCR.

So we have a pretty cool tool in the kit that allows us to flip an already deployed AP into an on-demand spectrum analysis collection client.  If I’m in my command centre in Calgary, and there’s a problem I need to troubleshoot at a site in Grande Prairie, I can look at the physical Layer 1 spectrum directly through the already installed infrastructure without even leaving my desk.  In my set-up, I can look in the Aruba AirWave management software I have and run historical reports on the RF health to match up to the spectrum analysis recording I took. Have fun!

What I learned from the CWAP

Having just challenged and successfully passed the recent version of the CWNP’s Certified Wireless Analysis Professional (CWAP-402) exam, I wanted to put some thoughts down on (virtual) paper about the experience. **DISCLAIMER** I’m not going to talk about, or even hint at the exam’s contents, you’ll have to book and experience it for yourself.

To start, what’s important is knowing who the Certified Wireless Network Professionals (CWNP) are, and why the program is so universally relevant.  From their website, https://www.cwnp.com/about  – “Certified Wireless Network Professional (CWNP) is the IT industry standard for vendor neutral enterprise Wi-Fi certification and training.”  The greatest part, for me, about the whole CWNP is the vendor-neutral part.  Talking to people about certifications I generally break it down this way; vendor technical certifications are typically focused on “how” to implement a solution or product X, which is good, has its place and is definitely important.  What the CWNP process offers though is the “why” and universal wireless truths.  Here’s a video breakdown of the learning track with the levels and disciplines: https://youtu.be/VuhONgb6rIM

cwnp-certs

For me, the CWAP exam represented the final of the three CWNP exams I needed to successfully challenge and complete before I could submit my application for the Certified Wireless Network Expert (CWNE).  It’s been a common comment and axiom in the wireless community that people should probably complete the CWAP first before the Design (CWDP) and Security (CWSP) Professional tracks. However, my job history and duties have dictated the order I’ve completed things.  Years ago, I was exclusively involved in the pre-sales and design phases, so learning the CWDP realm was most relevant.  Later, after a job change when I became fully engaged in the installation and operational phase, I had to take on a constant load of access control projects with Cisco’s Identity Services Engine (ISE).  This meant that for me at least, learning the CWSP material made the most sense.  So, like Keith Parsons (@KeithRParsons) says for a lot of wireless things, “it depends.”

Having said that, the CWAP, for me, was one of the most in-depth, challenging and interesting exams I’ve pursued.  When I say that, I don’t mean just the actual writing of the exam, I mean the study process as a whole.  If you’re studying for this exam, you must completely and totally commit and immerse yourself in the topics to get the most out of it.  Let’s look at the exam topics from the official CWNP website (https://www.cwnp.com/certifications/cwap):

  • 11 Physical (PHY) Layer Frame Formats and Technologies
  • 11 MAC Layer Frame Formats and Technologies
  • 11 Operation and Frame Exchanges
  • Spectrum Analysis and Troubleshooting
  • Protocol Analysis and Troubleshooting

When you read each of those five topics it’s very clear that you are going to be learning the foundations and nuts and bolts of the entire 802.11 process.  Not only that, but you have to know it cold, in detail and you will need to know how to use all the tools that test and troubleshoot the medium.

So now to the meat of it, what did I actually learn from the CWAP?  I definitely learned all the nuts and bolts of the Frame Control fields, not to mention the composition of the PHY Preamble and PLCP Header.  Wireshark filters for specific frame types and all the different frame types and subtypes in binary notation.  Oh, I filled two notebooks with facts, figures, sketches and scribblings.

cwap-books

But for me at least outside all the esoteric trivia I came away from the CWAP process with one lesson that is burned into my brain like never before:  how 802.11 wireless works.  I question the odds of me ever having to recall a great deal of the natty little details, especially if I can just look them up in reference materials, but when I put all those little bits together, I have a crystal-clear, unadulterated understanding of how and more importantly why 802.11 wireless works the way it does.  A good example of this would be with wireless Quality of Service, defined under the 802.11e amendment with Enhanced Distributed Coordination Access Function (EDCAF).  Studying and going through the CWAP process showed me exactly where in the 802.11 process QoS is applied, what exact parameters can be influenced and modified, how to troubleshoot to the nth degree, and most importantly, that it provides only probabilistic not absolute priority to clients.  For me that was huge, not just technically, but also for setting expectations with customers on what wireless QoS can actually accomplish.

Here’s a list of the core reference materials I used.  I also have to say that a home lab where you can capture wireless frames and analyze them in different scenarios is absolutely essential not just for this, but for any WiFi professional:

So here’s my advice to those who maybe just passed their CWNA and are trying to chart out their next step: study the CWAP content now.  Even if you aren’t going to challenge the exam, or are going to first challenge one of the other CWNP exams, or a manufacturer exam, pick the study guide(s) up and read them.  The core content within will help you with every single other piece of wireless learning you do. Enjoy the process, and good luck!

Designing for VoWLAN Part 1

Part of my job involves supporting my consultant peers in other cities/provinces with wireless projects.  For the most part, it means either assisting with wireless troubleshooting or doing the preliminary predictive designs and high/low-level design documents for a project.  One of these scenarios just came up, and I wanted to journal it, mostly for myself to look back on.

This particular customer is moving from their old offices into new buildings, four of them, and they are planning to leave the old WLAN infrastructure behind and start from scratch.  First few pieces of information I look for with any WLAN design is: scale floorplans to import in my Ekahau planning software, what are the client use cases and what physical devices they want to service with the new WLAN.  I find this is the basic starting point for most WLAN design efforts, kind of a wireless “design trinity.”

design-trinity

As anyone who’s done wireless design knows, getting what seems to be simple information from a customer can be challenging at best, and most of the time comes in dribs and drabs, and incomplete.  This case was no different; the floorplans had no marked scale, were PDFs, and were missing several floors, plus one whole building – encouraging start.  The use cases were fairly vague, but what did perk my ears up was that they wanted to use Skype For Business as an organization-wide collaboration tool.  Unfortunately, as I’m writing this, I still don’t know specifics on the actual client devices.  But if Skype is business critical, and I’m basing the design on the “neediest” use case, then I’m going to assume it’ll be running mostly on laptops and smartphones/tablets. In the absence of any other details, I send out a questionnaire and get the design started.  What I find is a good basis for any and all WLAN designs is captured in a past Ekahau webinar: http://www.ekahau.com/wifidesign/blog/2016/06/09/wi-fi-infographic-critical-aspects-tips-and-tricks-for-wlan-design/

The first thing I like to do when starting a WLAN design is I do an initial predictive design.  It gives me a general idea of AP quantities and types which are good for planning backend infrastructure like controllers and licensing, but even just drawing in the walls, aligning floors and shuffling APs around gives me a sort of “peace” where I can start visualizing what I’m going to want to look at during the validation, how I would configure things, test plans, the whole install strategy.  I use the Ekahau ESS Pro planning software, so with that fired up, and maps imported, I began to look at the design requirements and translating them into Ekahau-speak.  Lucky for me, Jussi, Mikko and the Ekahau crew have gift-wrapped WLAN design requirements for Skype already!:

skypeessrequirements

With these design requirements, making the predictive design “pass” wasn’t too tough.  The biggest test was meeting the Channel Overlap requirement.  To start, I made the decision to write-off the 2.4 GHz band for mission critical applications.  Just too many limitations, not the least are the limited number of non-overlapping channels.  5 GHz has far more frequency space to play in, so my first recommendation was to have any “corporate” SSIDs solely reside there, with 2.4 GHz reserved for guest or non-critical use.  With 5 GHz as the primary band, Channel Overlap and Co-Channel Interference (CCI) mitigation became a lot easier, but still one other decision to be made – channel width.  802.11ac gave us 80 MHz-wide and the theoretical 160 MHz-wide channels.  The problem is that every time you bond and double-up your channel width from the base 20 MHz to 40 MHz, then to 80 MHz, you reduce your number of usable, non-overlapping channels.  20-odd channels become 4 or 5 pretty quick.  With that in mind, I made the design decision to limit the 5 GHz channel width to the base 20 MHz only.  My rational was that it was likely I’d need a higher than normal density of APs to support the client use case (which is what this is about), and I would need the maximum number of channels in my reuse plan.

So I started placing my simulated APs in the predictive model, with constrained transmit power settings to control their cell size, and statically setting the operating channels to prove out a CCI-free environment was possible.  Success! (green means no calculated Channel Overlap)

cci-free

For the most part, the remaining exercise of the predictive design was tweaking to meet the design requirements like SNR and coverage.  One big, but often overlooked factor in designs, especially with high-roam apps like VoWLAN, is Secondary Cell coverage.  Primary cell coverage is pretty easy to achieve, but for VoWLAN, you’ll want to have a secondary cell coverage that will allow devices to gracefully roam as they move without dropping packets and cutting off conversations.  The trick I find is that when you start adjusting for solid secondary cell coverage, you can introduce CCI without meaning to.  This is where the tricky part came in, but in the end, I managed to get quality secondary RF coverage in the key areas where these users would be congregating, while maintaining a (theoretical) CCI-free environment.

secondarycell

So that was it for that phase, run the reports and write a nice summary to give all the heat maps some context.  The next steps are probably going to look like this (I hope):

Pre-Install Validation survey – unlikely to do a full APOAS survey, they seem like an inefficient use of time and effort given the tools and knowledge we now have.  At the very least, though, do some material attenuation testing, which IS important.  That predictive design I did, it’s predicated on some assumptions, foremost amongst them is the materials that the walls are made of and the attenuation effect they have.

Order, install and configure the gear – pretty straightforward, especially with a plan.  I’ve found that when you spend a lot of effort on a detailed low-level design, and Method of Procedure (MOP) for the AP installs, the actual installation phase can go very smooth.

Implement the business-critical clients and use-case – my recommendation was that the customer deploys wireless and offer a status quo situation as far as services go.  This will allow for a post-installation survey and remediation phase without disrupting important business processes.  It’ll also give time to do extensive client testing with the Skype application and the client devices running it.  A while back George Stefanick (@wirelessguru & @my80211) gave an amazing talk on the all-wireless office he and his team deployed, and one of the key lessons I got from it was how important exhaustive client testing is to the success of a wireless project. Check it out: https://www.youtube.com/watch?v=3RAyHNN_49Q&feature=youtu.be

So that’s it for now.  My hope is the next phases will fall into place as planned, and I’ll have a Part 2 of this post to write.

Sell Outcomes, Not Boxes

I like boxes I suppose. I appreciate the little ones APs come in for their efficiency in packing all the stuff into a small space.  The boxes that the WLCs come in are great for shipping Christmas and Birthday gifts across the country to family. As far as utility, a solid cardboard box has it.  Then you take a look at the boxes inside the boxes.  They’re making those new APs sleek and aesthetic and so, so pretty. Tiny, pretty boxes that push a million Jigabits now, with fewer calories and Full-Duplex with blinky lights.  For your server racks, we have boxes that fit in a standardized and uniform fashion – server and appliance boxes all lined up in an orderly, vertical row.

box

But I have never, ever had a customer call me and ask me to sell them boxes.  Never had a customer conversation go like this:

“Mr. Chappelle, it’s Mr. Smith at ABC Company.  Was wondering if you guys could sell us a couple hundred of those little WiFi WAP-thingy boxes, and oh, heck, throw in a couple of those big boxes, what do you call them?  Controllers?”

Doesn’t happen, ever.

What does happen is I get calls from customers and they are looking for an outcome, and they think that it has something to do with, or can be achieved with wireless technology.  Now I’m no expert on sales, but when someone has a problem and is looking for a solution, designing that solution towards an outcome is a lot easier to sell to a customer than just moving pallets of gear and boxes because they’re the newest, shiniest, <insert marketing bs>.  That’s not saying that upgrading, migrating and evolving to new technology should be discounted, not at all.  There are tons of new features, performance increases and functionality being added in the wireless space, and the rate of change is only growing.  But implementing this new gear and their new features has to support an outcome for the customer.  I recently read a tweet responding to the great @wirednot #WIFIQ and the great potential the upcoming 802.1ax amendment will be for WiFi.  The response basically said, “we don’t need faster than 802.11n performance, so not of impact.”  The point; more Jigabit-capable boxes would not support a positive business outcome for this customer.

Here’s another point to consider; many of us work in a system integrator or vendor environment that provides services/products other than just wireless.  Now, just think if you go to a customer and say, “hey, there are these new WAP-boxes that go super-uber-faster than the ones you have now, want to buy some?”  Then let’s say you sell them, and then swap out the old WAP boxes for the new WAP boxes.  Now on your way to the bank to deposit your commission cheque, the customer find out from a consultant, Gartner, or the good old Interwebs that the boxes you sold them really weren’t necessary for their business.  Short-term box-selling impulse just broke one of the hardest things to grow with a customer, trust.  That broken trust isn’t going to help down the road when maybe in six months they’re looking at moving to “the cloud,” or issue a storage RFP (they might not have issued if they trusted you).

The flip side to that is when you support and deliver an outcome for a customer, the trust, and long-term relationship only improve.  Real life example:  a customer approached me a while back, usual story “our WiFi sucks, guests complain constantly, we need new Wifi.”  The easy response would have been to just do a like-for-like swap out (more or less), replacing the old WAP-boxes for new ones, hung up in places I thought were better based on surveys, design, etc, etc.   Instead of just doing that, we sat down with the senior business leaders, took a long tour of the facility, took the time to understand their business and what they were looking for in an outcome, and then came up with a roadmap and strategy to transform their business through an infrastructure upgrade, including new wireless. The result was a successful project, tremendous customer satisfaction, and deal revenue five times what a mindless box replacement would have reaped.  It was a big win, for all sides, because the focus was on the outcome, not some boxes.

Dark Tale of Data Rate Pruning

It turns out, even in WiFi, there’s sometimes too much of a good thing.  Take Data Rates for example.  Wireless design best practices, the CWDP included, generally recommend disabling the lowest data rates in a wireless network.  We do this to prevent slow clients that can only support (for various reasons) the bottom data rates from slowing down all the other clients that are connected at higher performing rates – shared medium remember?  It’s become common for people to disable all the DSSS rates in 2.4 GHz (1, 2, 5.5 and 11 Mbps) to keep the ultra-damaging legacy 802.11b devices from crippling the network. So generally speaking, we design and implement modern wireless networks with data rates pruned somewhat to improve performance.

To set the stage a little, I have a customer that is a retailer, with stores all across Canada.  Their set-up on the wireless is pretty simple; Cisco hardware that’s a little long in the tooth, but serviceable.  They have Guest WiFi exclusively on 2.4 GHz and the Corporate and VoWiFi on 5 GHz only for simplicity. I got contacted recently because their Cisco 7925 wireless VOIP phones are “cutting out all the time and unusable.”  Lucky for me this is reported in the local store, so no airplane-time.  The contact asks me to go to site, because he suspects there are holes in the coverage, and he’d like me to walk around and survey the store for signal strength.  Since the store isn’t all that large, I do a quick walk around, gathering survey results in ESS and take a peek:

ess-capture

No real gaping holes in the coverage, maybe a little hot (we lowered the RRM Tx Power Max Threshold), but there isn’t anything that jumps out and says “hey that back stockroom has no signal.”  Time to look at the WLC I guess.

The customer had taken my advice a while back and had created separate RF Profiles for each store for flexibility.  Take a look at what I found when I went into the 802.11a profile specific to that store under the 802.11 tab:

datarate-pruning

I’m all for pruning data rates, but this seemed a little….extreme.  The disabled MCS rates aside, having Basic Data Rates disabled all the way to 48 Mbps can’t be a good thing.  Not for a client device that’s portable in the extreme, walking around a store with displays and shelving, and doors to go through, and all with a real-time VOIP call going on.  There’s bound to be a time that phone feels the need to rate shift because maybe the SNR just isn’t enough because a canoe or a human body got in the way. Without some lower data rates available, the phone just dropped its association and the call.

Sure enough, we enabled the data rates down to 24 Mbps, making it Mandatory, and the report back from the store a couple of weeks later was “working a million times better, no dropped calls.”  Lesson learned; most everything in moderation, and a little bit of thought before enabling or disabling something.

Warehouse Wireless – A Tale of Emergency RF Design Surgery

A few months ago I was engaged by an Account Manager to take a look at an issue that one of his customer’s was having in a warehouse space. Basically, the wireless coverage was so bad they couldn’t do their job: that was the diagnosis I was given, and it was up to me to fix it.

First step, discovery. How often do we talk to a customer or a sales person and get told “the WiFi doesn’t work,” and that’s the limit of the details we receive? So, I got on the phone and called the company ‘s IT manager to get the skinny on what was happening. This warehouse (in my hometown, Calgary) was the stocking distributor covering all of Canada for this company. They had a wireless LAN installed several years before, but “it never worked that well.” The users in the warehouse had two specific use-cases that relied on quality wireless: bar code scanners – Intermec model CK3’s, and Vocollect Talkman A720 Wireless Headsets. Office users used a different SSID for “data” coverage, but it wasn’t considered critical. Next step, establish a baseline and quantify the current RF environment through an onsite wireless site survey.

ck3

The warehouse itself wasn’t all that large, small by the standards of most. From a layout perspective, all the aisles ran in an East-West fashion, which would help with coverage planning. On the downside, these aisles were super narrow, maybe four feet across. Plus, the metal shelving almost reached the ceiling, thirty-five feet in the air. This made the deployment of their current WLAN even more challenging because the APs were mounted to the ceiling. There were four Cisco model 1242 APs suspended from the ceiling rafters, roughly a foot above the tops of the shelving. To make matters worse, the antennas selected were dipole, omni-directional types, suspended from the same rafters; and only servicing the 2.4 GHz radio band! From a very basic point of view, this deployment was doomed to fail. When I describe the RF propagation pattern of an omni antenna, I often say it looks like a donut on its side, and that its effective coverage area extends roughly twenty feet down. These antennas were: a)not going to reach the ground/users b)deployed in the wrong spot and c)only serviced one radio band. Breaking out my laptop and firing up my Ekahau Site Survey tool, I was able to confirm and quantify just how poor the coverage was, all due to bad design and equipment choices.

IMG_2607

Presenting the findings to the customer resulted in a solid plan of action: design a proper deployment, do a proof of concept and then deploy the planned solution. I like the opportunity to do a proof of concept, it is more or less an APOS survey, but it validates the plan not only for you as the engineer, but shows the customer that you know your stuff, and that the plan will work.

The design called for the following: replace the ceiling mounted, omni-antenna-equipped old APs with new, directional-patch antenna-equipped APs “shooting” RF down the aisles for pervasive coverage. Staying with Cisco infrastructure seemed to make the most sense; they had an existing investment in their controller and licensing, plus their other locations had Cisco WLANs, so Day 2 operational support could stay consistent. Cisco model 2702e Access Points and ANT2566 patch antennas were the gear chosen for the new install, but I needed some flexibility in mounting the antennas – a flat-wall mount wouldn’t work for me, I wanted to down-tilt the antennas slightly. @AccelTex and @elonsmitty to the rescue. AccelTex makes a wide range of accessories, like antenna mounts, for wireless installs that are a lifesaver.

IMG_0895

Installation day finally came, and all went according to plan. Once you get past the unique deployment model, the install is pretty stock. Cables were run and terminated, APs and antennas mounted, oriented and configured. There were some key configuration choices made, specifically to tailor the RF to the space and requirements. First off, 20 MHz channels in 5 GHz because of how many APs/antennas required, and the fact the DFS channels had to be disabled (next door to the airport literally). Enabling any sort of channel-bonding would have ruined the channel reuse plan and created unacceptable levels of Co-Channel Interference (CCI). The mission-critical users didn’t need high throughput, they needed reliable connections! Always remember who it is you’re serving, and what they need the wireless service for. Another crucial parameter that needed to be set was the Maximum Transmit Power threshold in the RRM section of Transmit Power Control (TPC). I know that a lot of people hate automatic RRM, but I find that within reason, and if setup properly, it is up to the task in the right situation. In this case, the default maximum of 30 dBm (which is insane IMHO), was lowered to a more reasonable 17 dBm which matches one of the 5 GHz power settings of the 2702e AP.

All that remained was the all-important post-installation validation survey and any fine-tuning. The post-install survey results looked very similar to the plan, which is exactly what you’re looking for. Because the design was optimized for the 5 GHz band, the mission-critical devices needed to move there. Fortunately, the scanners were smart enough to jump there automatically (with a little help from Band Select), but the “potatoes”, the A720 Voice Pods, needed to be manually configured to work on the 5 GHz band. Having a post-install survey report showing quality coverage throughout the facility is great for quantifying the success of the project, but hearing from the customer that “there hasn’t been a single ‘beep’ from us getting disconnected” is pretty rewarding and a good success measure too.