Bandwidth control is also known as Quality of Service (QoS) control. mbox appliances (HSG/CMG/HSA/UA) support extremely granular QoS control options, depending on the actual business requirements.
There’re several mechanisms to “control” bandwidth:
- Set maximum allowable bandwidth, this is often called “policing” or rate-limiting
- Guarantee minimum bandwidth for a particular network or application. This is part of our traffic-shaping feature
- Queue traffic based on different categories of traffic, and dequeue/release packets in order or priorities. This is often known as class-based queuing.
- Prioritize certain critical classes of traffic or application. In our mbox configuration, we simply need to put “prioritized” traffic on the highest queue (lowest class no.) in our traffic-shaping configuration.
There’re several ways to identify/mark interesting traffic for control:
- network-based, controls bandwidth (min and max) for a network or VLAN
- application-based, controls bandwidth (min and max) for a particular application
- host-based, controls bandwidth (min and max) on a per-host basis (it’s usually referred to as rate-limiting)
- user-based, controls bandwidth (min and max) for a particular user. A “user” can be identified by username or MAC address.
HSG, UA & HSA support additional bandwidth control features such as dynamic and adaptive bandwidth control.
Please refer to the below topology to explain the overall QoS concept. In the diagram, we have a backhaul of 1Gbps, and there are multiple internal VLANs. Each VLAN is typically mapped to a different SSID for wireless networks.
- Each VLAN is allocated with a bandwidth (we can set a min and max). The min guarantees a bandwidth, max sets how much this VLAN can burst when there’s available bandwidth in the whole pipe (eg. when other VLANs are not fully utilizing their min allocated bandwidth)
- For VLAN30, as an example, we use application-based QoS to further split and prioritize available bandwidth for different applications.
- For VLAN40, as an example, we use user-based QoS to offer differentiated services/experiences for different groups of users.
Class-based QoS basically allocates different types of traffic into different queues/classes, where you can also control the min/max bandwidth for each queue, and dequeues/releases packets out according to their priority (class no.).
It involves two simple steps:
- Traffic marking
- Traffic control (classification, prioritization, shaping)
Traffic/packet marking is a process to “color” the packets so that we can identify the targeted/”interesting” traffic which we want to control. Marking can be based on source/destination network/subnets, or protocol & port numbers (application).
Traffic marking is achieved using the firewall-set command with the “mark” option, eg.
firewall-set <rule-no> mark <marking-value> access src/dst/tcp/udp/dport/sport
the <rule-no> determinants the sequence of how firewall engines process each packet, matching a top-down process, once a packet is matched by a rule, it will not be processed further.
the <marking-value> basically sets a mark value for specific packets for the next processing. note this packet marking does not alter any packet header, meaning the marking does not follow a packet when it goes out, it’s “internal” within mbox for another process to refer (eg. classification, traffic shaping, etc).
the <src/dst/tcp/udp/dport/sport> sets matching criterion for the packets which we want to control. If we set a subnet/mask, we’re allocating bandwidth for a network; if we want to set bandwidth for applications, we use a combination of protocol and port numbers.
For example, if we want to allocate bandwidth for the entire network 172.16.1.0/24, configure the below rules to match the packets:
!firewall-set 10 mark 100 access src 172.16.1.0/24 remark “upload packets”firewall-set 11 mark 100 access dst 172.16.1.0/24 remark “download packets”!
If we want to do QoS for http application only, configure the below rules to match the packets:
! firewall-set 20 mark 100 access tcp dport 80 remark “user http requests” firewall-set 21 mark 100 access tcp sport 80 remark “server http replies" !
If we want to do QoS for http application only, configure below rules to match the packets:
! firewall-set 20 mark 100 access tcp dport 80 remark “user http requests” firewall-set 21 mark 100 access tcp sport 80 remark “server http replies" !
Traffic Control (classification, prioritization, shaping)
Classification basically arranges the colored packets into respective classes. A class identifies a network (VLAN), or type/group of application, depending on the marking criterion. All traffic with the same marking values belongs to the same class, and they’re assigned to their respective queue with desired bandwidth.
Each class has a class no (queue no), which sets the priority to dequeue the traffic when they leave the mbox exit interface. The lower class no. gets a higher chance to go out (higher priority).
mbox traffic shaping suppresses traffic when the link or traffic class gets congested. Unlike traffic policing, shaping method queues excess packets in a buffer (till the buffer is full) before dropping packets. This is a less “drastic” and smoother mechanism to manage traffic, compared with policing which simply drops excess packets, and mbox hardware usually comes with abundant memory/RAM to support a bigger buffer, making it an ideal solution for applications that are sensitive to packet drops.
We use the “traffic-shape” command to apply QoS control to the mbox exit interface.
! traffic-shape <min-link-bandwidth> <max-link-bandwidth> class <priority1> <min-class-bandwidth> <max-class-bandwidth> match fwmark <marking-value> class <priority2> <min-class-bandwidth> <max-class-bandwidth> match fwmark <marking-value> class <priority3> <min-class-bandwidth> <max-class-bandwidth> match fwmark <marking-value> !
traffic-shape only controls the traffic exiting the interface (egress traffic).
To control both upload & download, we need to apply duplicated rules (mirrored config) to both inbound and outbound interfaces.
The <min-link-bandwidth> & <max-link-bandwidth> is the total available bandwidth for the applied interface (eg. subscribed link speed).
Then we use class to further split different bandwidth for different groups of traffic.
If you are shaping traffic for different VLANs, the traffic-shape rule must be applied on the physical interface which the VLAN is tagged,
then use class to identify VLAN traffic. Do not configure traffic-shape on the VLAN interface.
If you are shaping traffic for a hotspot instance, the shaping rule must be configured with the hotspot instance (hotspot-shape)
Different classes can have different bandwidth allocation and prioritization.
<min-class-bandwidth> is the guaranteed bandwidth for this class. <max-class-bandwidth> is the maximum/ceiling bandwidth.
NOTE: each class bandwidth setting applies to all the host’s traffic within this class.
To accurately control the traffic, the sum of all classes <min-class-bandwidth> should not exceed <min-link-bandwidth>.
Traffic within a class can only burst to the <max-class-bandwidth>, even if there’s spare bandwidth within the entire link.
If we set the min and max to be the same, we’re effectively capping the bandwidth for a class. This is similar to policing/rate-limiting but in a smoother approach;
For critical classes (which we want them to burst as much as possible when there’s bandwidth available), we typically set <max-class-bandwidth> to be the same
as <max-link-bandwidth> so that they can burst up to fully utilize the link when there is spare (unused) bandwidth from other classes.
When a class’s actual bandwidth usage is less than <min-class-bandwidth>, the remaining bandwidth will be given to other needy classes which are allowed to
burst so that we can use up all available bandwidth. If multiple classes are bursting to grab the same spare bandwidth, they are “prioritized” based on the class
priority (eg. <priority1>, <priority2>, <priority3>)
- Determine the subscribed bandwidth for the link (WAN link speed).
- Determine which interface to apply traffic control (typically the WAN/outbound interface and LAN interface).
- Understand business requirements (eg. define traffic flows and assign them to respective classes and apply bandwidth for each class).
Example configuration (per VLAN QoS) steps
- Identify “interesting” traffic (as per step #1.1)
- Apply traffic-shaping on the target interface/link
- Set class priority and class bandwidth
- 1Gbps subscribed Internet link
- VLAN103 network: 10.0.103.0/24
- VLAN104 network: 10.0.104.0/24
- Assign VLAN103 network total bandwidth = 30Mbps
- Assign VLAN104 network total bandwidth = 40Mbps
Step 1: Traffic marking, identify the interesting traffic
! firewall-set 1031 mark 103 access src 10.0.103.0/24 remark VLAN103-UL firewall-set 1032 mark 103 access dst 10.0.103.0/24 remark VLAN103-DL firewall-set 1041 mark 104 access src 10.0.104.0/24 remark VLAN104-UL firewall-set 1042 mark 104 access dst 10.0.104.0/24 remark VLAN104-DL !
Step 2: Traffic control, on both inbound and outbound interface
! interface eth0 description “Interface to WAN” enable ip address 172.16.0.6/24 traffic-shape 1000000000 1000000000 class 103 30000000 30000000 match fwmark 103 remark “VLAN103 upload” class 104 40000000 40000000 match fwmark 104 remark “VLAN104 upload” ! interface eth1 description “Interface to LAN” enable traffic-shape 1000000000 1000000000 class 103 30000000 40000000 match fwmark 103 remark “VLAN103 download” class 104 40000000 40000000 match fwmark 104 remark “VLAN104 download” ! interface vlan 1 103 description "Wired Network vlan103" enable ip address 10.0.103.1/24 ! interface vlan 1 104 description "Wired Network vlan104" enable ip address 10.0.104.1/24 !
For download control, “traffic-shape” is applied on the physical LAN/eth1 interface, not each VLAN interface. VLAN103 and VLAN104 are attached to the eth1 physical interface. If we want to perform shaping for multiple VLANs, use “firewall-set” to mark packets from each VLAN subnets, and underclass bandwidth setting, set bandwidth for each VLAN based on the marking values.
While It is OK to use the same marking values for both inbound and outbound traffic for configuration simplicity, do note that each class control takes place at the exit interface (egress traffic). For example, for VLAN103 traffic, although we’re using the same marking value (103) for both upload and download in VLAN103, the underlying processing is not the same:
*on interface eth0, class 103 matches the packets marked by firewall-set 1031
*on interface eth1, class 103 matches the packets marked by firewall-set 1032
Step1. Check if packets are marked
mbox# show firewall set-list Chain PREROUTING (policy ACCEPT 15M packets, 14G bytes) pkts bytes target prot opt in out source destination Chain INPUT (policy ACCEPT 383K packets, 94M bytes) pkts bytes target prot opt in out source destination Chain FORWARD (policy ACCEPT 15M packets, 14G bytes) pkts bytes target prot opt in out source destination 417K 684M MARK all -- * * 10.0.103.0/24 0.0.0.0/0 /* set-list 1031 */ MARK set 0x67 348K 819M MARK all -- * * 0.0.0.0/0 10.0.103.0/24 /* set-list 1032 */ MARK set 0x67 454K 695M MARK all -- * * 10.0.104.0/24 0.0.0.0/0 /* set-list 1041 */ MARK set 0x68 329K 693M MARK all -- * * 0.0.0.0/0 10.0.104.0/24 /* set-list 1042 */ MARK set 0x68
Step2. Check if packets are matched and shaped
mbox# show interface traffic-shape
In our earlier section, we explained class-based QoS. Note that per class bandwidth refers to the total available bandwidth for that class, shared by all users within the network. Host-based control basically limits (or rate limiting) maximum bandwidth per IP host based on the configured policies.
Because some “bursty” users can exhaust the entire bandwidth and cause congestion to the class, especially during virus/storm outbreak, it is useful to be able to cap (or rate limit) per user or per host bandwidth so that all users or applications within the network can fairly share the available upstream pipe. Rate limiting is particularly important in public Wi-Fi networks, where attackers could use the free network (with huge backhaul) to launch DDoS attack against other victim networks.
To control per host bandwidth, we use the firewall-limit command. It is like a typical firewall rule but setting a bandwidth to this connection. This also means we can define very granular options to match a very specific host connection only.
! firewall-limit <rule_no> <max_bandwidth, Kbps> <direction> <src/dst> <Proto/Port or IP or subnet> !
*firewall-limit is also a type of firewall-access but it supersedes all other access lists. It’s also processed “top-down” based on the <rule_no>.
*There’s an implicit “permit” for each firewall-limit rule, eg. all traffic matching a limit rule will be permitted to pass through at the allowed <max_bandwidth> rate. Use “show firewall limit-list” to verify detailed policy of each configured rate-limit setting.
*If the rule defines src only, we typically are trying to limit upload; likewise if it defines dst only, we are limiting download.
*If we src/dst specifies a host IP only, this rule is just rate-limiting the particular host; if we define src/dst as subnet/mask, we are applying the same rate-limit to all hosts within the subnets.
*We can also limit per session bandwidth by combining src/dst/protocol (tcp/udp) and port numbers (eg, http/80 etc).
For example, if we configure the below rules, it means “each host within subnet 172.16.1.0/24 will be rate-limited to 2Mbps upload and download, for all applications, going through all interfaces.”
! firewall-limit 10 2048 all src 172.16.1.0/24 remark UL firewall-limit 11 2048 all dst 172.16.1.0/24 remark DL !
If we configure the below rules, it means “each host within subnet 172.16.1.0/24 will be rate-limited to 2Mbps upload and download, for http traffic only, going through all interfaces.”
! firewall-limit 10 2048 all tcp src 172.16.1.0/24 dport 80 remark UL-http firewall-limit 11 2048 all tcp sport 80 dst 172.16.1.0/24 remark DL-http !
It is possible to combine class-based QoS and Host-based QoS together. For example, we can set a total bandwidth for a network, then further rate limit per host bandwidth within that network.
In this example, we are trying to achieve the below objectives:
- subscribed backhaul bandwidth is 100Mbps
- network (LAN-eth1) is allocated total 50Mbps
- within LAN-eth1, per host is capped at 2Mbps upload/download
! interface eth0 description "Link to WAN/Internet" enable ip address dhcp traffic-shape 100000 100000 class 1 50000 50000 match fwmark 110 ! interface eth1 description "connection to LAN-eth1" enable ip address 172.16.1.1/24 dhcp-server dns 184.108.40.206 220.127.116.11 router 172.16.1.1 range 172.16.1.5 172.16.1.100 enable traffic-shape 50000 50000 class 1 50000 50000 match fwmark 110 ! ip name-server 18.104.22.168 22.214.171.124 ! firewall-set 10 mark 110 access src 172.16.1.0/24 remark Network-UL firewall-set 11 mark 110 access dst 172.16.1.0/24 remark Network-DL ! firewall-limit 10 2048 all src 172.16.1.0/24 remark "host UL" firewall-limit 11 2048 all dst 172.16.1.0/24 remark "host DL" ! firewall-snat 10 overload outbound eth0 !
Per user bandwidth control/allocation provides the granularity to assign bandwidth based on the user’s identity. This feature is only available when hotspot service is enabled (HSG and HSA). And this requires HSG/HSA to work with the local or external RADIUS server, where we can configure different user profiles, and attach the desirable access profile to specific users to map to our access policies, eg. user1 is assigned with 2Mbps upload/download, user2 can be assigned with 4Mbps upload/download.
*User accounts must be locally created within HSG RADIUS (or cloud RADIUS for HSA)
*User authorization profiles (access policies) must be created within the RADIUS server. Each access profile sets the min/max bandwidth per user.
*Each user must be attached with their respective access profile.
*Different users can have different access profiles, therefore having different access privileges even within the same network
- HotSpot service must be enabled on the user LAN, configured through CLI
- Configure user accounts, profiles, and assignments from the RADIUS portal
CMG and HSG support QoS at layer 2 by setting the class of service (CoS) for Ethernet frames. CoS (as in IEEE P802.1P) is a 3-bit field called the Priority Code Point (PCP) within an Ethernet frame header when using VLAN tagged frames as defined by IEEE 802.1Q. It specifies a priority value of between 0 and 7 inclusive that can be used by QoS disciplines to differentiate traffic.
In mbox, when configuring a VLAN interface, there’re two command options we can use
! interface vlan 0 10 enable set cos <default-outbound-priority> <mapped-priority> match cost <default-in-bound-priority> <mapped-priority> !
*set cos <outbound-priority> <mapped-priority>. This command rewrites priority for frames going out from the VLAN interface, so that the upstream switch can provide differential treatment (QoS) for the “colored” frames based on CoS value. By default, all frames have 0 (lowest priority, treat with best efforts). Set a target priority value <mapped-priority> for the upstream switch to use.
*match cost <inbound-priority> <mapped-priority>. This command does the reverse, for frames coming into mbox. By default, it rewrites all frames to 0, irregardless of the incoming frame priority. Use this command to preserve or change other values.
“firewall-set” command marks IP packet headers at layer 3/4 for the local or upstream routers to perform QoS control
“set” command sets priority bit on Ethernet frames at layer 2 for the upstream switch to perform QoS control
Some ISPs require to set CoS for the CPE WAN interface when trunking is configured. Below is an example of Cisco CoS config and mbox CoS config.
Cisco IOS config
! policy-map COS class class-default set cos 1 ! interface FastEthernet0.1103 encapsulation dot1Q 1103 ip address dhcp service-policy output COS !
! interface vlan 0 1103 enable set cos 0 1 ip address dhcp !