Thread: [HOWTO] Implementing Bandwidth Quota and Traffic Shaping on your Dedicated Seedbox

[Background]

If you have a dedicated server leased from OVH or Leaseweb, chances are it came with an upload bandwidth quota that varies depending on your signing plan. This could potentially be a problem if you have multiple users on this server where one or some users are bandwidth hog(s). Those users may be able to eat up all your months worth of bandwidth in a about a week or less. That wouldn't be fair for the rest of the users on the system.

This tutorial will help you appropriately* distribute the bandwidth among your users. We will accomplish this by using iptables and tc utilities which are installed by default on most current linux distributions. We will only care about the upload quota/shaping of each user. Note that this is by no means a comprehensive tutorial of ipfilter + tc. There are just too many combinations that we can use they won't all fit in this tutorial.

[Scenario]

A dedicated server connected to 100Mbps network that has a 4TB upload limit per month. There are 4 users of the seedbox: john, paul, george and ringo. Each user will then have 1 Terrabyte (1024KB^4) of monthly upload quota. We will not totally stop a user's connection once he reached his quota before the end of the next cycling period. If we do it like that then there wouldn't be any way for that user to connect to the server or rtorrent. What we're gonna do instead is to limit his upload connection to a stream of 2Mbps. Yes, he would still get pass his quota with this connection speed and count it against the overall upload limit. However, I sincerely doubt that he would be able to upload significant amount of data on that speed within a limited timeframe. But the point here is we still want that user to be able to do maintenance on his files via interactive ssh,sftp,ftp,http etc. Each user will have his own 2Mbps stream so he will not saturate other users bandwidth.

Additionally, we would like to exempt our bandwidth quota to IP addresses belonging to other OVH servers since internal OVH traffic is free.
Below are the list of OVH's IP Blocks:

IP RANGE : Netmask
87.98.128.0-87.98.255.255 : /16
91.121.0.0-91.121.255.255 : /16
94.23.0.0-94.23.255.255 : /16
188.165.0.0-188.165.255.255 : /16
213.186.32.0-213.186.63.255 : /19
213.251.128.0-213.251.191.255 : /18

[Commands]

First, I'm going to create a set of iptables and tc rules for user john. I will then explain each line individually.

root~:# iptables -t mangle -A POSTROUTING -o eth0 -p tcp -d 87.98.128.0/16 -j ACCEPT
root~:# iptables -t mangle -A POSTROUTING -o eth0 -p tcp -d 91.121.0.0/16 -j ACCEPT
root~:# iptables -t mangle -A POSTROUTING -o eth0 -p tcp -d 94.23.0.0/16 -j ACCEPT
root~:# iptables -t mangle -A POSTROUTING -o eth0 -p tcp -d 188.165.0.0/16 -j ACCEPT
root~:# iptables -t mangle -A POSTROUTING -o eth0 -p tcp -d 213.186.32.0/19 -j ACCEPT
root~:# iptables -t mangle -A POSTROUTING -o eth0 -p tcp -d 213.251.128.0/18 -j ACCEPT
root~:# iptables -t mangle -A POSTROUTING -o eth0 -p tcp -m owner --uid-owner john -m quota --quota 1099511627776 -j ACCEPT
root~:# iptables -t mangle -A POSTROUTING -o eth0 -p tcp -m owner --uid-owner john -j CLASSIFY --set-class 1:11
root~:# tc qdisc add dev eth0 root handle 1:0 htb default 99
root~:# tc class add dev eth0 parent 1:0 classid 1:1 htb rate 100Mbit ceil 100Mbit
root~:# tc class add dev eth0 parent 1:1 classid 1:11 htb rate 2Mbit ceil 2Mbit prio 2
root~:# tc class add dev eth0 parent 1:1 classid 1:12 htb rate 2Mbit ceil 2Mbit prio 2
root~:# tc class add dev eth0 parent 1:1 classid 1:13 htb rate 2Mbit ceil 2Mbit prio 2
root~:# tc class add dev eth0 parent 1:1 classid 1:14 htb rate 2Mbit ceil 2Mbit prio 2
root~:# tc qdisc add dev eth0 parent 1:11 handle 10: sfq perturb 10
root~:# tc qdisc add dev eth0 parent 1:12 handle 20: sfq perturb 10
root~:# tc qdisc add dev eth0 parent 1:13 handle 30: sfq perturb 10
root~:# tc qdisc add dev eth0 parent 1:14 handle 40: sfq perturb 10

Looking at the commands:

root~:# iptables -t mangle -A POSTROUTING -o eth0 -p tcp -d 87.98.128.0/16 -j ACCEPT
root~:# iptables -t mangle -A POSTROUTING -o eth0 -p tcp -d 91.121.0.0/16 -j ACCEPT
root~:# iptables -t mangle -A POSTROUTING -o eth0 -p tcp -d 94.23.0.0/16 -j ACCEPT
root~:# iptables -t mangle -A POSTROUTING -o eth0 -p tcp -d 188.165.0.0/16 -j ACCEPT
root~:# iptables -t mangle -A POSTROUTING -o eth0 -p tcp -d 213.186.32.0/19 -j ACCEPT
root~:# iptables -t mangle -A POSTROUTING -o eth0 -p tcp -d 213.186.32.0/19 -j ACCEPT

These rules tells to match outgoing packets destined to OVH network and let them through.

root~:# iptables -t mangle -A POSTROUTING -o eth0 -p tcp -m owner --uid-owner john -m quota --quota 1099511627776 -j ACCEPT

This rule tells to match all outgoing packets using the first ethernet and generated by john's process(es) that are destined to all other IP addresses, set a quota of 1TB and subtract bytes that are sent. If you have more than one ethernet card on your server you need to remove -o eth0 parameter on all iptables command.

root~:# iptables -t mangle -A POSTROUTING -o eth0 -p tcp -m owner --uid-owner john -j CLASSIFY --set-class 1:11

When john finally exhausted all his 1TB quota, we will throttle his upload speed to 2Mbps based on the tc class below.

root~:# tc qdisc add dev eth0 root handle 1:0 htb default 99

Create a root queuing discipline (qdisc) for eth0 and assign it a handle of (1:0). Define it as a hierarchial token bucket queue (htb) and assign a default queue class (99) for any classes not assigned to any class we are going to make. Note that this queue will also match virtual interfaces created from eth0 eg. eth0:1, eth0:2 etc.

root~:# tc class add dev eth0 parent 1:0 classid 1:1 htb rate 100Mbit ceil 100Mbit

Create a htb queue class (1:1) under root qdisc (1:0) and limit it's average and maximum stream rates to the rate of the interface (100Mbit). This is the parent class for all tc classes we will create.

root~:# tc class add dev eth0 parent 1:1 classid 1:11 htb rate 2Mbit ceil 2Mbit prio 2
root~:# tc class add dev eth0 parent 1:1 classid 1:12 htb rate 2Mbit ceil 2Mbit prio 2
root~:# tc class add dev eth0 parent 1:1 classid 1:13 htb rate 2Mbit ceil 2Mbit prio 2
root~:# tc class add dev eth0 parent 1:1 classid 1:14 htb rate 2Mbit ceil 2Mbit prio 2

First line means attach htb queue class to parent (1:1), with id (1:11) and limit the average stream rate to 2Mbit. Set the maximum rate equivalent to that of average rate. Give this class higher priority (the lower the number the higher the priority) so that ssh/sftp/http won't lag. We will only use this stream when john's bandwidth quota is equal to 0. The commands below it are for creating 2Mbit streams for the 3 other users.

root~:# tc qdisc add dev eth0 parent 1:11 handle 10: sfq perturb 10

Attach a qdisc handle to class 1:11. This line make sure that nothing hogs the pipe when the queue is saturated with packets.


To set quota and queuing of three other users, we only need to add two iptables rules.

paul:

root~:# iptables -t mangle -A POSTROUTING -o eth0 -p tcp -m owner --uid-owner paul -m quota --quota 1099511627776 -j ACCEPT
root~:# iptables -t mangle -A POSTROUTING -o eth0 -p tcp -m owner --uid-owner paul -j CLASSIFY --set-class 1:12

george:

root~:# iptables -t mangle -A POSTROUTING -o eth0 -p tcp -m owner --uid-owner george -m quota --quota 1099511627776 -j ACCEPT
root~:# iptables -t mangle -A POSTROUTING -o eth0 -p tcp -m owner --uid-owner george -j CLASSIFY --set-class 1:13

ringo:

root~:# iptables -t mangle -A POSTROUTING -o eth0 -p tcp -m owner --uid-owner ringo -m quota --quota 1099511627776 -j ACCEPT
root~:# iptables -t mangle -A POSTROUTING -o eth0 -p tcp -m owner --uid-owner ringo -j CLASSIFY --set-class 1:14

[Pitfalls]



There is one problem with our implementation though, Bandwidth quota will not work with http and sftp protocols. This is because Apache, Lighttpd and SSH all run under either 'root' or 'www-data' uid. We don't have a rule for those two users and we can't make one because ther is no way to account their bandwidth usage and have it subtracted to a certain user. What this means is that when, for example john, makes a connection to the web server and 'direct' downloads a 10GB file (which is counted as an upload for the server), that 10GB file will not be counted towards john's quota but rather to the server's overall 4TB monthly quota. 10GB would seem trivial compared to 4TB, however, if john starts downloading more files then we are losing a lot of our server's bandwidth quota. Moreover, if we try to trick the server to launch child process(es) under a user's id using third party plugins it will not work. All transfer connection streams are handled by the userid root or www-data.

The good news is, this problem does not exist in ftp/ftps. All child processes created by vsftp and proftpd that handles the transfer connection are passed to the userid who initiated the transfer. This is also not a problem in user created process such as rtorrent.

So, what to do? Well, I can't think of a better solution but to not provide sftp and http service at all to our users which is probably not a good solution. Maybe try to get the pid of that root process that handles the transfer connection by means of bash scripting? Sounds good, but it will add complexity to this tutorial. Or maybe, we can just limit http and sftp transfers and let the users use ftp/ftps instead for their downloads? Looks like it might work...
Let's try it...

Add this line to our POSTROUTING chain

root~:# iptables -t mangle -I POSTROUTING -p tcp -m multiport --sports 22,80,81,443 -o eth0 -j CLASSIFY --set-class 1:11

The above command means that we immediately shape ssh,http,https traffic to 2Mbps.


*** Remember that the 2Mbps upload stream will only kick in when the user connects via http(s), sftp and once the user's quota has been reached. Otherwise, everyone's upload speed on rtorrent/ftp/ftps or any other user created process defaults to 100Mbps ***

[TESTING]

Now we're up to some testing using screenshots.

This is what our POSTROUTING table should look like without the OVH IP blocks. As you can see, john has an initial quota of 1099511627776 bytes (1TB).


We then download a 600+MB file using ftp. We can see that we are utilizing the bandwidth at full speeds.


Next, we check back the iptables status. We find the quota is now 677,000,000+ bytes less.


We try sftp. Download speeds are down to average 2-3Mbps.


http direct download. We still have an average of 2-3Mbps.


We go back to look at iptables and find that john's quota, after downloading using sftp and http, did not change.


[MAINTENANCE]

Another thing we have to worry about is system reboot. Because iptables and tc rules are deleted during reboot, we need to find a way to be able to save/restore those quota counters. Fortunately, there is a command for that. iptables-save and iptables-restore.

Before we reboot we need to save the tables and its byte counters.

root~:# iptables-save -c > /etc/iptables.save
root~:# reboot

Restore the tables and byte counters after booting. Unfortunately, there is no way to save tc rules for now. So, we need to re-type all the 'tc' directives for every user.

root~:# iptables-restore -c /etc/iptables.save
root~:# tc qdisc ... ...
root~:# tc class ... ...

We can also create a crontab entry that saves the tables every 5 minutes. This is also good in an event of a system crash.

root~:# crontab -e

Add the following line:
*/5 * * * * /sbin/iptables-save -c > /etc/iptables.save

Save and exit cron

[Quota Reset]

Every month we need to reset the quota. This will depend on your billing cycle date (Check with OVH Manager). We do this by deleting all iptables rules then re-creating them.

Delete iptables rules, then re-create the rules (see [Commands]).

root~:# iptables -t mangle -v -L --line-numbers
root~:# iptables -t mangle -D POSTROUTING N (where N is the line number, on the left side, we want to delete)

We don't have to reset our tc entries but it's always a good practice to reset it as well, then re-create rules.

root~:# tc qdisc del dev eth0 root

Additionally, we can do this in cron if you know the exact date and hour every month of your next OVH monthly quota reset.

First, we create a simple bash script.
root~:# nano ~/reset-quota
--------- Enter code below ---------------

PHP Code:

#!/bin/bash -e
ipt=/sbin/iptables
tcc=/sbin/tc
a=$($ipt -t mangle -v -L --line-numbers | cut -d' ' -f1 | tail -1)
for ((i=a; i>0; i--))
do
$ipt -t mangle -D POSTROUTING $i
done
$tcc qdisc del dev eth0 root 


--------- End -----------------------------
Save and exit file
root~:# chmod 750 ~/reset-quota

then we create a crontab entry that will be executed every 5th day of the month at 12PM.
root~:# crontab -e
* 12 5 * * /root/reset-quota
Save and exit crontab

Of course, you need to re-create all iptables/tc rules again.

Also, we can remove all bandwidth limitations whenever it's convinient for us. Let's say there's an overall 1TB of bandwidth left 2 days before our next month's billing cycle ends. We would like to have everyone else to use that remaining bandwidth.

root~:# crontab -e
* 12 3 * * /root/reset-quota
Save and exit crontab

[STATUS]

To view the iptables status we run the command:

root~:# iptables -t mangle -v -L --line-numbers

To view tc current defined qdisc and classes

root~:# tc qdisc show dev eth0
root~:# tc class show dev eth0

[ADDITIONAL NOTES]

I will be updating this thread and upload a bash script to automate the creation of rules. I don't have ETA but I will upload it sometime when I get another free time. If you have any questions, just ask and I'll try to answer them.

[SOURCES]

This tutorial are all based on my own research but it wouldn't be possible without the help of the following links, specially the first two:
Iptables: http://linux.die.net/man/8/iptables
tc (This utility is hard to find documentation): http://linux.die.net/man/8/tc
Some helpful bandwidth limiting using iptables and tc sites:
http://www.linuxquestions.org/questi...letely-592827/
http://www.linuxquestions.org/questi...x-user-744365/
http://www.linuxquestions.org/questi...r-user-828874/
Bit Calculator: http://www.matisse.net/bitcalc/?inpu...otation=legacy

WOW, this tutorial is absolutely amazing, this is something I was looking all over the net and I can't believe that now I have the chance to give it a try..
Awesome work, in next couple of day I will try your suggested method and many thanks for posting it!

root~:# iptables -t mangle -A POSTROUTING -o eth0 -p tcp -d 87.98.128.0/16 -j ACCEPT
root~:# iptables -t mangle -A POSTROUTING -o eth0 -p tcp -d 91.121.0.0/16 -j ACCEPT
root~:# iptables -t mangle -A POSTROUTING -o eth0 -p tcp -d 94.23.0.0/16 -j ACCEPT
root~:# iptables -t mangle -A POSTROUTING -o eth0 -p tcp -d 188.165.0.0/16 -j ACCEPT
root~:# iptables -t mangle -A POSTROUTING -o eth0 -p tcp -d 213.186.32.0/19 -j ACCEPT
root~:# iptables -t mangle -A POSTROUTING -o eth0 -p tcp -d 213.186.32.0/19 -j ACCEPT

Hi scoobie,

With the commands above will the traffic only pass to an OVH boxes and that way lets say I have 5 TB bandwidth I would be able at least double it to 10TB because the traffic only pass to an OVH boxes?

Originally Posted by iSMAEL

Hi scoobie,

With the commands above will the traffic only pass to an OVH boxes and that way lets say I have 5 TB bandwidth I would be able at least double it to 10TB because the traffic only pass to an OVH boxes?

What those commands mean are to just pass and not process the uploaded packets to those ip ranges and not account them in our quota rules.

It's not really relaying the packets we sent so they will be sent out to the final destination outside OVH network. It just happened that if you have leechers from servers within those IP ranges, we won't account them in our quota counter.