How QoS Works in the Backend

Overview

On the UTM, the term Quality of Service (QoS) encompasses a variety of technologies to shape network traffic. You configure QoS to look for specific types of traffic and how to handle it, for example, prioritize or limit it.

The UTM chooses the most appropriate technology to achieve the result you want. In general, the Linux tc command (short for traffic control) is used for this. Traffic control is very complex which is why this article only provides a pragmatic look at it from the point of view of a UTM administrator.

Basic Operation

In WebAdmin, you enable QoS on an interface and set the total bandwidth on an interface. For example, if you have a T1 connection, you configure 1.581 Mbit/s. This tells the UTM that all traffic should not exceed 1.581 Mbit/s.

Then, you configure a traffic selector. It tells the UTM what kind of traffic to look for, for example, any DNS traffic.

You then create a bandwidth pool to tell the UTM what to do with the traffic selector. For example, for the DNS traffic selector, you guarantee 100 kbits/s of your total bandwidth.

Filters and iptables

The actual implementation of QoS rules in the backend uses a combination of iptables and tc filters.

When a packet enters the UTM, iptables evaluates it and determines whether the packet is destined for the UTM itself or a device behind it. iptables is able to modify packets on a binary level ("mangle"). For example, if a packet matches a certain criterion, iptables can alter some information of the packet.

In terms of QoS, if iptables finds a match of a specific type of traffic (as configured in the WebAdmin traffic selectors), it changes an IP header field to contain something called a "handle". Next, tc takes the handle and matches it to a specific class (see "HFSC" below). After that, tc applies its internal logic to shape the traffic.

Commands

To view iptables rules in the backend, access your UTM through SSH and run the following command:

utm:/root # iptables -nvL QOS_AUTO_PRIO -t ips
Chain QOS_AUTO_PRIO (2 references)
pkts  bytes target   prot opt in out source    destination
934   48736 CLASSIFY tcp  --  *  *   0.0.0.0/0 0.0.0.0/0   tcp flags:0x17/0x02 -m length2 --layer4 --length 20:40 CLASSIFY set 2:1
12998 564K  CLASSIFY tcp  --  *  *   0.0.0.0/0 0.0.0.0/0   tcp flags:0x15/0x10 -m length2 --layer4 --length 20:40 CLASSIFY set 2:1
30    4587  CLASSIFY tcp  --  *  *   0.0.0.0/0 0.0.0.0/0   tcp spts:1:65535 dpt:53 CLASSIFY set 2:1
1093  87436 CLASSIFY udp  --  *  *   0.0.0.0/0 0.0.0.0/0   udp spts:1:65535 dpt:53 CLASSIFY set 2:1

Here's an explanation of the command line output from iptables:

tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp
spts:1:65535 dpt:53
Indicates the traffic selector parameters.
CLASSIFY set 2:1
Action that iptables will take when this rule matches (set a parameter in the IP header).
tcp flags:0x15/0x10
Flag that is set, to be referenced by tc (set 0x15 to a value of 0x10).

To view tc rules in the backend, access your UTM through SSH and run the following command:

utm:/root # tc filter show dev eth0
filter parent 1: protocol all pref 1 fw
filter parent 1: protocol all pref 1 fw handle 0x2711 classid 1:20
filter parent 1: protocol all pref 1 fw handle 0x2710 classid 1:1

Here's an explanation of the command line output from tc:

parent 1
Indicates which class this filter applies to.
protocol all
Indicates which protocol (tcp, udp, icmp, etc...) this filter applies to.
fw handle 0x2710
Indicates which flag matches this filter.
classid 1:10
Indicates which class to use for a packet that matches this filter.

Now you know how the UTM filters the traffic.

Queue Disciplines

How the UTM knows what to do with identified packets.

A Queue Discipline (qdisc) is an algorithm that determines how to handle traffic shaping. There are several algorithms. Here, you get to know a few of them and how they relate to the UTM.

pfifo_fast

This is the default queue discipline for Linux. The acronym fifo is a common computing term meaning "first in, first out". In the context of QoS this means that the bytes (packets) that enter the QoS subsystem are queued. When dequeued, the bytes that entered first are the ones that leave first. The discipline uses 3 distinct queues, also known as priorities (0, 1 and 2).

In the TCP header for all TCP packets, there is a byte reserved for type of service (TOS) information. Depending on the value of this TOS byte, pfifo_fast queues the packet in queue 0, 1 or 2. pfifo_fast empties queue 0 before queue 1, and once the queues 0 and 1 are empty, it empties queue 2. If not set already, pfifo_fast sets the TOS byte to 0x0 and puts the packet into queue 1.

In WebAdmin, this queue discipline comes into play automatically if another device set the TOS byte. You can also set TOS bits under Network > Quality of Service (QoS) > Traffic Selector. If the byte is set, the UTM prioritizes the traffic and when sent to the next hop, the TOS byte remains. (What happens after that is not in the control of the UTM.)

WebAdmin limits what you can set for the TOS byte. In theory, you can use any byte value (hexadecimal 0-f).

Using the command tc qdisc, you see a sequence of 16 digits that can be either 0, 1 or 2. This sequence is the priority map. The digits specify the queue to use for a packet with a given TOS value. Starting from left to right, the value increments (0x0, 0x1, 0x2 ... 0xf). Here's an example of an output:

utm:/root # tc qdisc
pfifo_fast 0: dev eth1 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1

Note that qdisc uses pfifo_fast in this output. This output is for dev eth1. You should have at least one pfifo_fast qdisc for each configured interface.

The table below shows the detail of the priority map. Compare your WebAdmin configuration to this to see which configurations are available and how they are used in the backend.

WebAdmin Value Normal Service Minimize Monetary Cost Maximize Reliability N/A Maximize Reliability N/A N/A N/A Maximize Throughput N/A N/A N/A N/A N/A N/A Minimize Delay
Hex 0x0 0x1 0x2 0x3 0x4 0x5 0x6 0x7 0x8 0x9 0xa 0xb 0xc 0xd 0xe 0xf
Binary 0000 0001 0010 0011 0100 0101 0110 0111 1000 1001 1010 1011 1100 1101 1110 1111
Queue 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1

HFSC

This queue discipline uses an algorithm called Hierarchical Fair Service Curve (HFSC). Hierarchical refers to the algorithm using classes. Bandwidth allocation takes place similar to a tree structure: Under full operation, HFSC can use a full tree with child nodes and leaf nodes. The UTM only uses leaf nodes. In WebAdmin, you configure the total bandwidth for the parent node under Network > Quality of Service (QoS) > Status. You then configure the guaranteed and limited bandwidth on the Bandwidth Pools tab.

For more information on HFSC, go to: http://linux-ip.net/articles/hfsc.en/

To view HFSC configurations on the command line, access your UTM through SSH and run following command:

utm:/root # tc class show dev eth0
class hfsc 1: root
class hfsc 1:1 parent 1: sc m1 0bit d 0us m2 102400Kbit ul m1 0bit d 0us m2 102400Kbit
class hfsc 1:10 parent 1:1 leaf 100: sc m1 0bit d 0us m2 1024Kbit ul m1 0bit d 0us m2 102400Kbit
class hfsc 1:20 parent 1:1 leaf 200: sc m1 0bit d 0us m2 100000bit ul m1 0bit d 0us m2 250000bit
class hfsc 1:30 parent 1:1 leaf 300: sc m1 0bit d 0us m2 101276Kbit ul m1 0bit d 0us m2 102400Kbit

The table shows an explanation of the most important fields:

1:[x[x]]
Indicates the hierarchy. 1: is root. 1:1 is the parent. 1: followed by 2 digits is the leaf node. HFSC also supports mode nodes, but the UTM doesn't use these.
parent 1:1	
Indicates which parent to use. It references the hierarchy indicator.
leaf xxx
Assigns a leaf number to be used elsewhere.
first bit value
The guaranteed bandwidth. Should see m2 before and u1 after this field.
second bit value
The limit bandwidth. This is the final field in the line.

Note that if you didn't specify an upper limit, HFSC uses the total bandwidth as upper limit.

How To Use tc To Diagnose Connections

With the tc command you can use the option -s for stats and -d for details.

Show queue disciplines:

tc [-s|-d] qdisc [dev eth#]

Show classes:

tc [-s|-d] show classes dev eth#

View a real-time list of stats (updated every 0.2 seconds):

watch -n .2 'tc -s qdisc'

Sample output of the above command is below. Troubleshooting with this method is similar to watching iptables counters: You can see which rules are being used:

Every 0.2s: tc -s qdisc
qdisc ingress ffff: dev eth0 ----------------
Sent 844798 bytes 11284 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps
backlog 0b 0p requeues 0
qdisc hfsc 1: dev eth0 default 30
Sent 959520 bytes 1801 pkt (dropped 0, overlimits 405 requeues 0) rate 0bit 0pps
backlog 0b 0p requeues 0

From the tc man page (https://man7.org/linux/man-pages/man8/tc.8.html).

DEV is the interface name, for example, eth0:

tc qdisc [ add | change | replace | link ] dev DEV [ parent qdisc-id | root ] [ handle qdisc-id ] qdisc [ qdisc specific parameters ]
tc class [ add | change | replace ] dev DEV parent qdisc-id [ classid class-id ] qdisc [ qdisc specific parameters ]
tc filter [ add | change | replace ] dev DEV [ parent qdisc-id | root ] protocol protocol prio priority filtertype [ filtertype specific parameters ] flowid flow-id
tc [-s | -d ] qdisc show [ dev DEV ]
tc [-s | -d ] class show dev DEV
tc filter show dev DEV

Examples

Default config with no QoS rules

Three physical interfaces and one SSL VPN interface

utm:/root # tc qdisc
qdisc pfifo_fast 0: dev eth2 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth1 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth0 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev tun0 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
utm:/root # tc class show dev eth1
utm:/root #
utm:/root # iptables -nvL QOS_AUTO_PRIO -t ips
Chain QOS_AUTO_PRIO (0 references)
pkts bytes target prot opt in out source destination
utm:/root # tc filter show dev eth0
utm:/root #

With one QoS rule

On eth1, guarantee 1024 kbit/s for DNS. Note that there are two defined traffic selectors, but only one is used.

karlsruheutm:/home/login # tc qdisc
qdisc pfifo_fast 0: dev eth2 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc hfsc 1: dev eth1 root refcnt 2 default 3
qdisc fq_codel 803d: dev eth1 parent 1:2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
qdisc prio 2: dev eth1 parent 1:3 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc fq_codel 803e: dev eth1 parent 2:1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 803f: dev eth1 parent 2:2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 8040: dev eth1 parent 2:3 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
qdisc ingress ffff: dev eth1 parent ffff:fff1 ----------------
qdisc hfsc 1: dev eth0 root refcnt 2 default 2
qdisc prio 2: dev eth0 parent 1:2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc fq_codel 8041: dev eth0 parent 2:1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 8042: dev eth0 parent 2:2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 8043: dev eth0 parent 2:3 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
qdisc pfifo_fast 0: dev tun0 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc tbf 1: dev ifb0 root refcnt 2 rate 95Mbit burst 57Kb lat 4295.0s
qdisc fq_codel 10: dev ifb0 parent 1:1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
qdisc tbf 1: dev ifb1 root refcnt 2 rate 972800Kbit burst 596812b lat 4295.0s
qdisc fq_codel 10: dev ifb1 parent 1:1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
utm:/home/login # tc class show dev eth1
class hfsc 1: root
class hfsc 1:1 parent 1: sc m1 0bit d 0us m2 100Mbit ul m1 0bit d 0us m2 100Mbit
class hfsc 1:2 parent 1:1 leaf 803d: sc m1 0bit d 0us m2 1024Kbit
class hfsc 1:3 parent 1:1 leaf 2: sc m1 0bit d 0us m2 98976Kbit
class fq_codel 803d:37c parent 803d:
class prio 2:1 parent 2: leaf 803e:
class prio 2:2 parent 2: leaf 803f:
class prio 2:3 parent 2: leaf 8040:
class fq_codel 803f:13b parent 803f:
utm:/root # iptables -nvL QOS_AUTO_PRIO -t ips
Chain QOS_AUTO_PRIO (2 references)
pkts  bytes target   prot opt in out source    destination
934   48736 CLASSIFY tcp  --  *  *   0.0.0.0/0 0.0.0.0/0   tcp flags:0x17/0x02 -m length2 --layer4 --length 20:40 CLASSIFY set 2:1
12998 564K  CLASSIFY tcp  --  *  *   0.0.0.0/0 0.0.0.0/0   tcp flags:0x15/0x10 -m length2 --layer4 --length 20:40 CLASSIFY set 2:1
30    4587  CLASSIFY tcp  --  *  *   0.0.0.0/0 0.0.0.0/0   tcp spts:1:65535 dpt:53 CLASSIFY set 2:1
1093  87436 CLASSIFY udp  --  *  *   0.0.0.0/0 0.0.0.0/0   udp spts:1:65535 dpt:53 CLASSIFY set 2:1
utm:/root # tc filter show dev eth1
utm:/root #

With two QoS rules:

On eth1, guarantee 1024 kbit/s for DNS, and guarantee a minimum of 100 kbits and a maximum of 250 kbits for port 9100.

karlsruheutm:/home/login # tc qdisc
qdisc pfifo_fast 0: dev eth2 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc hfsc 1: dev eth1 root refcnt 2 default 4
qdisc fq_codel 8044: dev eth1 parent 1:2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 8045: dev eth1 parent 1:3 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
qdisc prio 2: dev eth1 parent 1:4 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc fq_codel 8046: dev eth1 parent 2:1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 8047: dev eth1 parent 2:2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 8048: dev eth1 parent 2:3 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
qdisc ingress ffff: dev eth1 parent ffff:fff1 ----------------
qdisc hfsc 1: dev eth0 root refcnt 2 default 2
qdisc prio 2: dev eth0 parent 1:2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc fq_codel 8049: dev eth0 parent 2:1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 804a: dev eth0 parent 2:2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 804b: dev eth0 parent 2:3 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
qdisc pfifo_fast 0: dev tun0 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc tbf 1: dev ifb0 root refcnt 2 rate 95Mbit burst 57Kb lat 4295.0s
qdisc fq_codel 10: dev ifb0 parent 1:1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
qdisc tbf 1: dev ifb1 root refcnt 2 rate 972800Kbit burst 596812b lat 4295.0s
qdisc fq_codel 10: dev ifb1 parent 1:1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
utm:/home/login # tc class show dev eth1
class hfsc 1: root
class hfsc 1:1 parent 1: sc m1 0bit d 0us m2 100Mbit ul m1 0bit d 0us m2 100Mbit
class hfsc 1:2 parent 1:1 leaf 8044: sc m1 0bit d 0us m2 1024Kbit
class hfsc 1:3 parent 1:1 leaf 8045: sc m1 0bit d 0us m2 100Kbit ul m1 0bit d 0us m2 250Kbit
class hfsc 1:4 parent 1:1 leaf 2: sc m1 0bit d 0us m2 98876Kbit
class fq_codel 8044:333 parent 8044:
class prio 2:1 parent 2: leaf 8046:
class prio 2:2 parent 2: leaf 8047:
class prio 2:3 parent 2: leaf 8048:
class fq_codel 8047:e3 parent 8047:
class fq_codel 8047:2cc parent 8047:
utm:/root # iptables -nvL QOS_AUTO_PRIO -t ips
Chain QOS_AUTO_PRIO (2 references)
pkts  bytes target   prot opt in out source    destination
934   48736 CLASSIFY tcp  --  *  *   0.0.0.0/0 0.0.0.0/0   tcp flags:0x17/0x02 -m length2 --layer4 --length 20:40 CLASSIFY set 2:1
12998 564K  CLASSIFY tcp  --  *  *   0.0.0.0/0 0.0.0.0/0   tcp flags:0x15/0x10 -m length2 --layer4 --length 20:40 CLASSIFY set 2:1
30    4587  CLASSIFY tcp  --  *  *   0.0.0.0/0 0.0.0.0/0   tcp spts:1:65535 dpt:53 CLASSIFY set 2:1
1093  87436 CLASSIFY udp  --  *  *   0.0.0.0/0 0.0.0.0/0   udp spts:1:65535 dpt:53 CLASSIFY set 2:1
utm:/root # tc filter show dev eth1
utm:/root #

More Information

Linux Advanced Routing & Traffic Control How-to: https://tldp.org/HOWTO/Adv-Routing-HOWTO/

tc man page: https://man7.org/linux/man-pages/man8/tc.8.html

Good HFSC primer: http://linux-ip.net/articles/hfsc.en/