System commander Script for troubleshooting Fortinet Infrastructure AP reboot/radio reset issues

Its sometime get tough troubleshooting an AP Reboot issue. While troubleshooting Fortinet WiFi APs, You might need to capture the AP flash event log to identify the cause of reboot. However, some time AP flash space ran out  already storing Maximum logs files(This could happen if AP is Up and running log time in production or there are many events happening in AP that filled up the space soon). In this case you might need to clear those flash log space, so it can store the latest/ fresh log files.

This is needed because you making sure that those are latest logs of the crash/reboot happened.

#You run the following command on AP to clear the old flash logs:

file areaerase logk0 

file areaerase logk1

#You run the following command to read the flashlogs in the AP:

flashcmds show logk0

flashcmds show logk1

However, Incase of big deployments where you cannot run this on each AP separately and  also without causing WIFI service disruption you have to run this at-least on a bunch of AP to save your time. We have something called “system commander” where you will be executing a script to clear those AP flash logs.

Step1:

To use the system commander tool you have to enable telnet on the controller.
FortiWLC# configure terminal

FortiWLC(config)# telnet enable

Step2:

Please create a folder on the c drive of your computer and copy these two files shared on the link below:
https://www.dropbox.com/s/82ir713iegxj0tt/SystemCommandExecutor_Ver14.exe?dl=0

https://www.dropbox.com/s/otrkiobxmhnhq4y/SystemCommands_AP.txt?dl=0

Step3:
Run the following commands from your computer command prompt window.

SYNTAX: cfolder_name SystemCommandExecutor_Ver14.exe -c ipaddressofcontroller-username-password -o 4 -a APMODEL -l 1

Example for AP822s: 

C:\PRAVEEN>SystemCommandExecutor_Ver14.exe -c 192.168.10.3-admin-admin -o 4 -a AP822 -l 1

Explanation:

The folder name is the name of the folder in which you have saved the files(SystemCommands_AP.txt) on c drive. The username and password are the login credentials of the controller. The logs will be saved in the same folder on c drive once the loop is completed.

If you look at the syntax it has AP model mentioned in that so please choose the AP model accordingly, depending on for which AP you want to collect the logs.
* For any and all AP reboots we need to run this to gather AP reboot logs (file areaerase logk0/logk1)
* After collecting Disable Telnet

1. On the command prompt we need to specify AP model. Since we can only collect with particular AP type. If customer has mixed AP environment we need to collect separately for each AP model.
2. If we have any AP which is disable offline. Make sure to delete that entry before running System commander, since loop will end at that point and we don’t have option to start again from there. (Ex: If we have 100 AP on network and AP ID 44 is offline, loop will end at AP Id 44)

Addon :You can add more commands on the script file(SystemCommands_AP.txt) to execute and get to collect a dump of those command output’s too.

 

 

Single Channel Architecture VS Multi Channel Architecture

This Blog post is about Fortinet Single Channel Architecture(legacy MERU) and Multi Channel Architecture. This blog is nothing related to the discussion on which will outperform the other.

Since I work on both SCA and Multi channel architecture, Just thought of sharing some insight about what is all this Single Channel Architecture and how that you can compare with that of traditional MCA.

 

Single Channel Architecture Multi-Channel Architecture

 

1 Wireless System in control of WLAN network by deciding the Client Association and Roaming Factors Wireless clients do make decisions in your WLAN network for Association and Roaming
2 Wireless clients are connected on single channel called VIRTUAL CELL and their Random back-off algorithm “CWmin – CWmax” for clients  in the same contention space are proposed by Wireless System.

Pros:#No Channel Planning required and easy administration.#Client’s Contention space is decided/Managed by wifi system. So even if there are other client sending/receiving data in same channel the system will calculate, if sending or receiving data from any other client on a calculated RSSI going to cause any impact for others and then decision will be made to give transmit opportunity.

Cons: Design flaws can cause:*high Channel usage*High AP Neighbor count*Management overhead

Wifi clients are allowed to run and select independent BACKOFF value(CWmin-CWmax) for channel access.Note:Since individual clients choose their  own backoff algorithm and no coordination between clients result in retry and Channel access collision mostly in High density environment. So You are forced to create small Cells.Pros:#Effective Spatial reuse(channel planning)and TX power operation is must but once done it works reliably#Can say MCA works parallel to RF physics

Cons: Design flaws can cause:*CCI and ACI*TX power and channel plan*Client contention causing Retries and    Collision when number of client increased.

3 Airtime Fairness is TIME based fairness Even MCA vendors doing Time based Airtime Fairness for long time now.
4 Recommended EIRP 2.4Ghz: 17dbm && 5Ghz: 23dbmNote: Might need to reduce TX power based on Deployments(considering the signal propagation.i.e, More open space env. Recommended EIRP 2.4ghz:10dbm  && 5ghz: 12dbm(+ or – 3dbm).

 

5 Number of Visible Access point for clients
Data = 2 or 3 APs
Number of Visible Access point for clients
Data = 2 APs
6 VHD=20MHZ, HD=40MHZ,Many SCA Deployments do have channel width of 80MHZ set and works pretty well.  Note sure about field values for 80+80(160)MHZ VHD=20MHZ, HD=20 and/or 40MHZCannot think about 80 MHZ or 80+80(160)MHZ in enterprise network.

 

7 In High density deployments you might need to check Duty Cycle because management frame and beacons are heard from other APs crowded on Same spacial Diversity(channel), Look for Retry and AP Neighbor counts because that could cause potential issues.Note: Refer High Density Design Guide In High Density Environment you might need to still bring down the TX power/disable low data rates consideration of Antenna Type, ie: Yagi or patch or omni

 

8 High Density SCA network With More client/bandwidth Requirement  needs Channel Layering or Channel striping and AP poding  and that depends on the Building Structure.  High density/capacity requirement needs you to add more radio and form small micro cells with least CCI by spacial reuse and either disabling radio or reducing AP TX power and more focused coverage pattern and etc.
9 Virtual Cell to Virtual Cell roaming is HARD HANDOFF.Since Microcell is also supported in Fortinet Infrstucture OS, I would like to mention Microcell to microcell is efficient with PMK, 802.11.                                                                Microcell to Microcell roaming is supported efficiently by PMK,OKC,802.11KVR.Note: No virtual cell support.

 

10 Supports both SCA and MCA with ARRP(Automatic radio resource provisioning. Supports MCA only with (Automatic radio resource provisioning).

 

My take on SCA over MCA:

SCA “Virtual cell” method take on the traditional “Micro cell” method by negotiation of RF overhead(How well can manage) vs Clients contention based Retry/Collision(How well can manage)

And the System has a central brain call “coordinator” which reads the RF condition based on the interim updates from different supporting worker modules, takes decision in providing better AIRTIME Fairness and user performance for wireless clients.

Test Bed:

If you would like to play around with Fortinet-SCA more just setup your test environment for the following and you can come up with your test cases and values.

A simple example that is easily replicated in the lab is to take five or six access points, all configured for the same channel, and 30 or so wireless clients. Put them all in one room and observe the aggregate throughput.

Then take those same clients and access points and distribute them across the floor of a building and observe the new, increased/decreased aggregate throughput.

 

 

My First Wireless community meet APAC

This Blog post just comes out of excitement, Guess who i meet ?

“Keith R. Parsons” founder of  WLPC and  a CWNE.

“Ronald van Kleunen”  Board of advisors for CWNE Certifications and a CWNE himself.

I could keep telling more about what they mean for this WIFI community and how much they have done for the community.

I always wanted to meet them personally and ask my questions regarding the Current Industry Wireless Design goods and bad with Keith and regarding Wireless Security with Ronald.  And I didn’t had to wait for too long , yes i got this chance on 05/05/2017 @ Singapore.

During the day of session 1:

we had a brief discussion about Wireless design practices and  keith shared a little of  his design experiences . Topics including  how the Wireless signal modulation works, whats with Multipath and how does a client behave in relation to those in RF physics and clients NIC cards algorithm and more about the WiFi Designs on real world.

Some notes from the session:

  • Sin, -Sin, Cos, -Cos explanation about how Wi-Fi devices communicate using “Symbols”

        (and SGI – Short Guard Interval = 400ns and non-SGI =  “Long” Guard Interval =          800ns)

  • BPSK (IEEE 802.11 1Mbps, 1-bit per symbol), QPSK (IEEE 802.11 2Mbps, 2-bits per symbol)

and all the way up to 256-QAM (.11ac, 8-bits per symbol).

keith

Session 2 of the Day:

My favorite,  Security and Wireless briefed by Ronald it was so much of information than I expected. How much is security important in your Wireless Infrastructure and what if we don’t meet and who are those organization and government body takes care of theses audit. And what if we don’t follow and to what extent the penalty would be.

Topics covered:

*Vendor neutral education

*Secure the Wireless LAN infrastructure

24*x7 Monitoring and Reporting Wireless Security (Automation) using Wireless Intrusion Prevention

*Systems (WIPS) and Mobile Device Management (MDM) Systems

*Regularly do wireless security audits (wireless vulnerability assessments) in the organisation

Later session:

We had a CWNEs round table. CWNEs are really open to guide other WIFI peers and happy sharing the knowledge for the community. They did discuss about their CWNE career and how it is helping them to do their job better.

I have to thank Ronald for organizing this meet and efforts in bring APAC Wireless community together.

CWNEs  onboard / Round table : Keith, Ronald, Ram, Jeffrey, Shuang

Finally this happened, I got lucky on that day and I did receive an Awesome bag in Lucky Draw 🙂

Lastly, WirelessLAN Professionals Badge from Keith Made My Day.

802.11 Wireless frame Aggregation and A-MSDU and A-MPDU in 802.11 N/AC

802.11 Frame Aggregation 

The Frame aggregation was introduced in purpose of reducing the overhead on wireless space and by doing so increasing the efficiency and performance of wireless client. By introducing frame aggregation in the given amount of  Airtime/TXOP the wireless client could now send more data than before.

MSDU-MAC Service Data Unit 

The upper layer information for a MAC layer in 802.11 is called the  MSDU. i.e the stuff [IPheader|TCP-UDP header|Data].

MPDU-MAC Protocol Data Unit

Once there is a header and footer added to the MSDU its now called as MPDU. This is now send to the lower layer called physical layer.

MSDU

Aggregated-MSDU

This is new function added from 802.11N this make it possible to send more data  during the given Transmit opportunity. Collection of MSDU packets is called A-MSDU and then sending them in a single burst through the physical layer.

a-msdu

Aggregated-MPDU:

Again this is new function added from 802.11N and its collection on multiple MPDUs and send them in a single burst through physical layer. You will find multiple mac header and mac checksum inside a A-MPDU.

a-mpdu

Maximum 802.11 Frame Body Size(without encryption)

MSDU: 2304 bytes [standard frame]

MPDU: 2346

MSDU inside A-MPDU: 4065 bytes

A-MSDU: 7955  bytes

802.11AC @11,454

Key points on this discussion:

* 802.11AC the transmission is always aggregated (a-msdu or a-mpdu) even though its a small chunk of data to be transferred and its backward compatible.

*Since the maximum MTU size on a wired network is 1500bytes this is further getting fragmented on wired network (When Jumbo frames disabled on wired).

AirCapture with MacBook

Sniff AirTraffic using Macbook is easy and you just have to follow the steps below:
1.)Quit all open applications.
2.)Try to join the Wi-Fi network that you are having issues with if you are not already connected.
3.)Open Wireless Diagnostics. Tip: You can hold down the Option key and then click the Wi-Fi menu extra.
4.)Enter your admin name and password when prompted.
5.)In Wireless Diagnostics, choose Window > Sniffer,
5.)Choose appropriate channel and channel width as per your Wireless AP Radio configuration

Start the capture , click stop when you are done , A capture file will be placed on the desktop.

I also found a Video done my benmiller regarding the same:

http://www.sniffwifi.com/2013/10/how-to-capture-wifi-free-in-mac-os-x.html

COA | Change Of Authorization

COA | Change of authorization
RFC 3576
UDP port 3766 | IEEE

>Change of authorization is used to ask NETWORK SWITCH/ROUTER/WIFI-CONTROLLER to disconnect-user, session timeout, bounce port of the existing user session on network.
>Mostly AAA server is user to record the user accounting information and based on the user activity or time the user is then disconnected from the network or session timed out.
>The Network device will send the accounting information to a AAA server.
>The AAA server keeps track of user information like Device MAC-ADDRESS, User SESSION-ID, USERNAME, FRAMED-IP-ADDRESS and more based on what it gets from accounting request.
>The radius accounting request traffic will be send only after a successful authentication.
>You could configure to send interim accounting update about the user to accounting server.This interim update can help in scenarios where you want to disconnect the user after usage of certain amount of bandwidth in network. Default time:300 Sec

#Things to remember while working on a COA setup:
>Make sure the network device is configured to except the COA request from COA server on respected port.
>NAS devices wants to see particular attriutes in COA request to identify the user session and perform COA, So sending the right attributes on COA request is important and it depends on the NAS device vendor.
>The expected COA user identification attribute might be different for a Captive portal authenticated user and Dot1.x user.
>Some vendor ignote the unsupported or unknown attributes in COA request and still ACK your COA request. However some vendor devices dont like you sending a unsupported COA attributes, so they respond with disconnect-NAK
#Here in case of Fortigate Firewall you might need to send the right COA supported attribute to identify the user session.

*supported attributes: WPA-Enterprise&UserGroup and Captive Portal supports:
USER_NAME,
FRAMED_IP_ADDRESS,
EVENT_TIMESTAMP,
MESSAGE_AUTHENTICATOR,

*User-name” and “Frame-ip” were supported in DM request and both MUST be involved, other attributes like “Calling-Station-Id”, “Called-Station-Id” could not be supported and would cause 503 error message.
*The attributes “EVENT_TIMESTAMP” and “MESSAGE_AUTHENTICATOR” are options.
*Note: Supported attributes as on the latest firmware v5.4

#In case of ARUBA MOBILIY CONTROLLER:

The mobility controller supports the following attributes for identifying the users who authenticate with an RFC 3576 server:

* user-name: Name of the user to be authenticated.
* framed-ip-address: User’s IP address.
* calling-station-id: Phone number of a station that originated a call.
* accounting-session-id: Unique accounting ID for the user session

–> Other attribute might cause 503 error.

#Other ports to Remember:
*Radus Accounting request and Response is on UDP1813
*Radius Authentication is on UDP1812

#Some snapshots attached here to see what’s there inside Disconnect-request,Disconnect-,ACK,Disconnect-NAK.

Accounting Request:

radius-acc

Disconnect Request:

Disconnect-request

Disconnect-ACK:

Disconnect_Ack

Disconnect-NAK:

Disconnect_NAK

If would like to look at the entire pcap, let me know I can share with your guys.

 

Separating your RF space for priority traffic with Single Channel Architecture and Channel Layering

Segmentation of services can be achieved by doing channel layering with Fortinet Infrastructure WiFi(SCA). By doing so you actually separate  business critical traffic  from  your non critical and Guest traffics.

There are applications like VO-WIFI and few other low latency sensitive application that need special care taken.

#Wired infrastructure :You deploy a separate Vlan and end-end QOS written and so.

#On WIFI Controller you can do inbound/outbound QOS

#The WMM supported clients have TX-OP /Access category  Q for prioritizing the voice/video traffic.

In a SCA you further get the opportunity to do Segmentation of services by creating  a separate RF space for them.

Ref Pic:

SCA