Guide The Linux Series! [PART 3]

[Trance]

Description

This topic is part of a multi-part series. We'll try to get everything straight to the point in this guide, without unnecessary over-explanation.
 

PART 1 [CLICK HERE]

• Work faster with a better terminal emulator.
• Use a better editor.
• Basic L2J server setup.
• Manage and secure your MySQL server. [!]

 

PART 2 [CLICK HERE]

• Secure your Linux server.
• Tuning system profiles. [!]
• Network performance tuning. [!]
• How to build and manage a firewall using iptables and conntrack - simplified version. [!]

 

PART 3 [THIS GUIDE]

• Understanding and managing the OVH Firewall. [!]
• How to build and manage a firewall using iptables, conntrack, ipset and synproxy - advanced version. [!]
• Mitigating most of the DDoS attacks. [!]

Understanding and managing the firewall at OVH

 

 

Please read carefully as I am sharing the intellectual property with you right now.

OVH will NOT vacuum any traffic unless it is detected by them that you are being attacked. In fact, the firewall is disabled by default and OVH will force it to enable once they have detected that you are being attacked. This can take anywhere from a few seconds to a few minutes, no VAC in the world will be triggered instantly. I think some of the strongest VACs in the world fire in about 12 seconds.

If your machine is vulnerable, it may fill its resources before OVH's VAC can help you. That is why it is very important to have a machine on Linux that is well configured.

I will say one more thing that will probably blow your mind. OVH's VAC is not perfect, in fact it has very few vulnerabilities, one of the vulnerabilities is that it cannot block malformed ACK connections - which we'll handle it on our Linux server.

Create a Firewall

In the OVHcloud Control Panel, click on the Bare Metal Cloud menu and open IP. You can use the drop-down menu underneath “My public IP addresses and associated services” to filter your services according to category.

Configuring the Network Firewall

A priority (from 0 to 19, 0 being the first rule to be applied, followed by the others).

0. Same networking rule applies to the OVH Firewall, like we did with the IPTables firewall. Allow all ESTABLISHED. If the connection is already ESTABLISHED with the server, it will ignore all of the below.

 

1, 2. Allowing my VPN networks for any TCP/IPV4 protocol - ICMP (ping-ing), TCP, etc; for any port. You can do the same for your webhost if you have anything that needs to connect to your MySQL server; you could limit it to TCP and 3306 port though instead of accessing it all.
 

3, 4, 5. Allowing any TCP connection on the specific ports our server needs. It's very important to have the option SYN checked. So the attacker can't use our ports if a connection is not ESTABLISHED.

When a new player is connecting to the server, it will go to Rule 3 and 4 and then any other communication further will be handled by Rule 0 because it's already ESTABLISHED, meaning any other rule after that point won't apply.



The OVH Firewall won't deny connections by default, so we have to add a rule at the end - Rule 19. To deny it all. 
 

From the same drop-menu where we've enabled and configured the Firewall, we can activate Mitigation: permanent mode. So we have it triggered at all times for a faster response.

How To Build And Manage A Firewall with IPTables, Conntrack, IPSet and SYNPROXY

We will get more serious now. Keep in mind that we'll be working with the IPTables rules we've created in the previous topic, so do NOT jump that part!

 

We will still use IPTables and Conntrack, and on top of that we'll use IPSet and SYNPROXY.

IPSet

We'll use it to create sets of addresses to allow or block with our IPTables rules.
We will manually add the IPs we want to allow them access to certain ports for management purposes.
We'll also use it to automatically keep IPs that iptables will block for a certain amount of time that we set with IPSet.

P.S. They're part of the installation packages in one of the previous parts of my Linux Series.

We'll create several sets:

ipset_allowed_ports will be used to keep the trusted IPs we need to access SSH, MySQL, etc.
 

ipset create ipset_allowed_ports hash:net

ipset_whitelist will be used to manually whitelist ourselves to ignore every single IPTables rule; for extra precaution. This will the first rule in every singe table/chain.
 

ipset create ipset_whitelist hash:net

ipset_players_whitelist will be used to manually whitelist Players that somehow are blocked by our rules. This would only happen if they're traffic is suspiciously triggering our Anti-DDoS rules. It usually happens if they're infected with a malware or something like that. It happened to about 5 players on my previous servers in the past. Hey! We care about all our players.
 

ipset create ipset_players_whitelist hash:net

ipset_blacklist will be used to manually ban whoever we want.
 

ipset create ipset_blacklist hash:net

The next 3 sets will be used by the iptables' rules to add banned IPs for 300 seconds. Preferably to increase it to several hours once it has been tested.
 

ipset create banned_limit_conn hash:net timeout 300
ipset create banned_limit_synproxy_ack hash:net timeout 300
ipset create banned_limit_syn hash:net timeout 300

Tips and tricks:

See all ipsets:
 

ipset list

Flush all IPs in a set:
 

ipset flush banned_limit_conn
ipset flush banned_limit_synproxy_ack
ipset flush banned_limit_syn

Manually add an IP to our sets example:
 

ipset add ipset_allowed_ports 188.0.0.193 # My VPN #2
ipset add ipset_players_whitelist 94.71.4.115 # Player "ANU"
ipset add ipset_whitelist 188.0.0.193 # My VPN #2

SAVE IPSET
 

ipset save > /etc/ipset.conf

Clean the config so you can save it all again - assuming you did some changes.
 

sudo echo -n "" > /etc/ipset.conf

IPSET SERVICE

By default, CentOS won't store those sets to be used at startup.
We'll have to create a service dedicated for it instead.

Let's create the service file by suing our favorite editor nano.

 

sudo nano /etc/systemd/system/ipset-persistent.service

 

ipset-persistent.service:
 

[Unit]
Description=ipset persistent configuration
Before=network.target

# ipset sets should be loaded before iptables
# Because creating iptables rules with names of non-existent sets is not possible
Before=netfilter-persistent.service
Before=ufw.service

ConditionFileNotEmpty=/etc/ipset.conf

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/sbin/ipset restore -exist -file /etc/ipset.conf
# Uncomment to save changed sets on reboot
# ExecStop=/sbin/ipset save -file /etc/iptables/ipset
ExecStop=/sbin/ipset flush
ExecStopPost=/sbin/ipset destroy

[Install]
WantedBy=multi-user.target

RequiredBy=netfilter-persistent.service
RequiredBy=ufw.service

 

And save it by using CTRL+X, Y, ENTER.

Now we need to run the following commands to reload daemon, restart the service we've created, check the status and enable it at startup.
 

systemctl daemon-reload
systemctl stop ipset-persistent.service
systemctl start ipset-persistent.service
systemctl status ipset-persistent.service
systemctl enable ipset-persistent.service

SYNPROXY
 

Linux Kernel is vulnerable to simple SYN attacks. The basic TCP scalability problem for the Linux kernel is related to how many new connections can be created per second. This directly relates to a lock per socket when in the "listen" state. For "established" state connections, it can scale very well. The "listen" state lock is encountered not only with SYN packets, but also other initial connection state packets like SYN-ACK and ACK packets (the last three-way handshake (3WHS) packet). In the flooding attack scenario, the attacker is sending fake packets aimed at hitting the "listen" state locking problem. As such, we need a mechanism to filter out these fake initial connection attempts before the socket enters the "listen" state lock and blocks new incoming connections.

With SYNPROXY, the result is a 20x increase in deflecting SYN-ACK and ACK based attacks.

We'll have to use IPSet alongside SYNPROXY.

Let's get started by creating the needed logs. Basically, when a packet needs to be dropped by iptables' rules, it will instead log them and add them into the ipsets. This way we have full control, and thanks to IPSet we don't have to add a rule for each IP we want to filter.
 

iptables -t mangle -N log_limit_conn
iptables -t mangle -A log_limit_conn -m limit --limit 1/second --limit-burst 5 -j LOG --log-prefix "banned_limit_conn:drop: " --log-level 4
iptables -t mangle -A log_limit_conn -j SET --add-set banned_limit_conn src
iptables -t mangle -N log_limit_synproxy_ack
iptables -t mangle -A log_limit_synproxy_ack -m limit --limit 1/second --limit-burst 5 -j LOG --log-prefix "banned_limit_synproxy_ack:drop: " --log-level 4
iptables -t mangle -A log_limit_synproxy_ack -j SET --add-set banned_limit_synproxy_ack src
iptables -t mangle -N log_limit_syn
iptables -t mangle -A log_limit_syn -m limit --limit 1/second --limit-burst 5 -j LOG --log-prefix "banned_limit_syn:drop: " --log-level 4
iptables -t mangle -A log_limit_syn -j SET --add-set banned_limit_syn src

 

Now we can use either the MANGLE or RAW table to block the unwanted connections before reaching other chains. We'll use the RAW table.
 

iptables -t raw -A PREROUTING -m set --match-set ipset_whitelist src -j ACCEPT # Whitelist
iptables -A INPUT -m set --match-set ipset_whitelist src -j ACCEPT # Whitelist
iptables -A OUTPUT -m set --match-set ipset_whitelist src -j ACCEPT # Whitelist
iptables -t raw -A PREROUTING -m set --match-set ipset_blacklist src -j DROP # Blacklist
iptables -t raw -A PREROUTING -m set --match-set banned_limit_conn src -j DROP # Limit concurrent conn per IP
iptables -t raw -A PREROUTING -m set --match-set banned_limit_synproxy_ack src -j DROP # # Ratelimit the ACK from 3WHS handled by SYNPROXY
iptables -t raw -A PREROUTING -m set --match-set banned_limit_syn src -j DROP # Limit SYN

 

Adding ipset_players_whitelist into the MANGLE table before our banned ipsets. So it will avoid those rules but not the rest due to security risks.
 

iptables -t mangle -I PREROUTING -m set --match-set ipset_players_whitelist src -j ACCEPT

Adding then we add the following rule AFTER the rules FROM THE PREVIOUS PART OF THE SERIES.
 

iptables -t mangle -I PREROUTING -m set --match-set ipset_players_whitelist src -j ACCEPT

Here we start the implementation of the SYNPROXY in the RAW table.

1. SYNPROXY works on untracked conntracks, it will create the appropriate conntrack proxied TCP connections. Also, we want it to only be used on the enp3s0f0 interface which is my NIC with the public IP. Yours may be named eth0 or eth1.

 

iptables -t raw -A PREROUTING -i enp3s0f0 -p tcp -m tcp --syn -m multiport --dports 2106,7777 -j CT --notrack

P.S. You can check your work interfaces via ifconfig example:
 

enp3s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 51.10.10.10  netmask 255.255.255.255  broadcast 51.10.10.10
        inet6 fe80::f816:3eff:fe4e:b469  prefixlen 64  scopeid 0x20<link>
        ether fa:16:3e:4e:b4:69  txqueuelen 1000  (Ethernet)
        RX packets 15063483  bytes 3025357817 (2.8 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 14028825  bytes 2853497188 (2.6 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 5872280  bytes 674391486 (643.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 5872280  bytes 674391486 (643.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

2. Now we add some extra rules to help our SYNPROXY even more - to use the 3 ipsets banned we added above.
This is getting extra clever about it where you won't find anything similar in a mix with SYNPROXY on the internet.
 

iptables -t mangle -A PREROUTING -i enp3s0f0 -p tcp -m tcp --syn -m multiport --dports 2106 -m connlimit --connlimit-above 5 --connlimit-mask 32 --connlimit-saddr -j log_limit_conn
iptables -t mangle -A PREROUTING -i enp3s0f0 -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK ACK -m multiport --dports 2106,7777 -m conntrack --ctstate INVALID -m hashlimit --hashlimit-above 5/sec --hashlimit-burst 10 --hashlimit-mode srcip --hashlimit-srcmask 24 --hashlimit-name limit_synproxy_ack -j log_limit_synproxy_ack
iptables -t mangle -A PREROUTING -i enp3s0f0 -p tcp -m tcp --syn -m multiport --dports 2106,7777 -m hashlimit --hashlimit-above 5/sec --hashlimit-burst 10 --hashlimit-mode srcip --hashlimit-name limit_syn --hashlimit-srcmask 24 -j log_limit_syn

 

1st rule will limit concurrent connections per IP, currently only set on the Login Server 2106, so you still let Internet Cafes and multiple game boxes.
2nd rule will ratelimit the ACK from 3WHS handled by SYNPROXY.
3rd rule will limit the SYN, which we discussed above why that is important with SYNPROXY.

3. Now going to the FILTER table, we finish the implementation of the SYNPROXY.

 

iptables -A FORWARD -i enp3s0f0 -p tcp -m tcp -m multiport --dports 2106,7777 -m state --state INVALID,UNTRACKED -j SYNPROXY --sack-perm --timestamp --wscale 7 --mss 1460
iptables -A FORWARD -i enp3s0f0 -p tcp -m tcp -m multiport --dports 2106,7777 -m state --state INVALID -j DROP

1st rule is catching state: UNTRACKED == SYN packets, INVALID  == ACK from 3WHS.
2nd rule will drop rest of state INVALID, this will e.g. catch SYN-ACK packet attacks.

This will protect from 99% of the DDoS attacks if your resources can handle it.

 

A recap with all rules put together in the proper order:

 

### [IPSET] ###
ipset create ipset_allowed_ports hash:net
ipset create ipset_whitelist hash:net
ipset create ipset_players_whitelist hash:net
ipset create ipset_blacklist hash:net
ipset create banned_limit_conn hash:net timeout 300
ipset create banned_limit_synproxy_ack hash:net timeout 300
ipset create banned_limit_syn hash:net timeout 300

### [LOGS] ###
iptables -t mangle -N log_limit_conn
iptables -t mangle -A log_limit_conn -m limit --limit 1/second --limit-burst 5 -j LOG --log-prefix "banned_limit_conn:drop: " --log-level 4
iptables -t mangle -A log_limit_conn -j SET --add-set banned_limit_conn src
iptables -t mangle -N log_limit_synproxy_ack
iptables -t mangle -A log_limit_synproxy_ack -m limit --limit 1/second --limit-burst 5 -j LOG --log-prefix "banned_limit_synproxy_ack:drop: " --log-level 4
iptables -t mangle -A log_limit_synproxy_ack -j SET --add-set banned_limit_synproxy_ack src
iptables -t mangle -N log_limit_syn
iptables -t mangle -A log_limit_syn -m limit --limit 1/second --limit-burst 5 -j LOG --log-prefix "banned_limit_syn:drop: " --log-level 4
iptables -t mangle -A log_limit_syn -j SET --add-set banned_limit_syn src

### [RAW] ###
iptables -t raw -A PREROUTING -m set --match-set ipset_whitelist src -j ACCEPT # Whitelist
iptables -t raw -A PREROUTING -m set --match-set ipset_blacklist src -j DROP # Blacklist
iptables -t raw -A PREROUTING -m set --match-set banned_limit_conn src -j DROP # Limit concurrent conn per IP
iptables -t raw -A PREROUTING -m set --match-set banned_limit_synproxy_ack src -j DROP # # Ratelimit the ACK from 3WHS handled by SYNPROXY
iptables -t raw -A PREROUTING -m set --match-set banned_limit_syn src -j DROP # Limit SYN
iptables -t raw -A PREROUTING -i enp3s0f0 -p tcp -m tcp --syn -m multiport --dports 2106,7777 -j CT --notrack # SYNPROXY works on untracked conntracks, it will create the appropiate conntrack proxied TCP conn

### [MANGLE] ###
iptables -t mangle -A PREROUTING -m set --match-set ipset_whitelist src -j ACCEPT # Whitelist
iptables -t mangle -A PREROUTING -p tcp ! --syn -m conntrack --ctstate NEW -j DROP # Drop TCP packets that are new and are not SYN
iptables -t mangle -A PREROUTING -p tcp -m conntrack --ctstate NEW -m tcpmss ! --mss 536:65535 -j DROP # Drop SYN packets with suspicious MSS value
iptables -t mangle -A PREROUTING -p tcp --tcp-flags ACK,FIN FIN -j DROP # Block packets with bogus TCP flags
iptables -t mangle -A PREROUTING -p tcp --tcp-flags ACK,PSH PSH -j DROP # Limit SYN
iptables -t mangle -A PREROUTING -p tcp --tcp-flags ACK,URG URG -j DROP
iptables -t mangle -A PREROUTING -p tcp --tcp-flags FIN,RST FIN,RST -j DROP
iptables -t mangle -A PREROUTING -p tcp --tcp-flags SYN,FIN SYN,FIN -j DROP
iptables -t mangle -A PREROUTING -p tcp --tcp-flags SYN,RST SYN,RST -j DROP
iptables -t mangle -A PREROUTING -p tcp --tcp-flags ALL ALL -j DROP
iptables -t mangle -A PREROUTING -p tcp --tcp-flags ALL NONE -j DROP
iptables -t mangle -A PREROUTING -p tcp --tcp-flags ALL FIN,PSH,URG -j DROP
iptables -t mangle -A PREROUTING -p tcp --tcp-flags ALL SYN,FIN,PSH,URG -j DROP
iptables -t mangle -A PREROUTING -p tcp --tcp-flags ALL SYN,RST,ACK,FIN,URG -j DROP
iptables -t mangle -A PREROUTING -s 224.0.0.0/3 -j DROP # Block spoofed packets
iptables -t mangle -A PREROUTING -s 169.254.0.0/16 -j DROP
iptables -t mangle -A PREROUTING -s 172.16.0.0/12 -j DROP
iptables -t mangle -A PREROUTING -s 192.0.2.0/24 -j DROP
iptables -t mangle -A PREROUTING -s 192.168.0.0/16 -j DROP
iptables -t mangle -A PREROUTING -s 10.0.0.0/8 -j DROP
iptables -t mangle -A PREROUTING -s 0.0.0.0/8 -j DROP
iptables -t mangle -A PREROUTING -s 240.0.0.0/5 -j DROP
iptables -t mangle -A PREROUTING -s 127.0.0.0/8 ! -i lo -j DROP
iptables -t mangle -I PREROUTING -m set --match-set ipset_players_whitelist src -j ACCEPT # Players Whitelist
iptables -t mangle -A PREROUTING -i enp3s0f0 -p tcp -m tcp --syn -m multiport --dports 2106 -m connlimit --connlimit-above 5 --connlimit-mask 32 --connlimit-saddr -j log_limit_conn # Limit concurrent conn per IP
iptables -t mangle -A PREROUTING -i enp3s0f0 -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK ACK -m multiport --dports 2106,7777 -m conntrack --ctstate INVALID -m hashlimit --hashlimit-above 5/sec --hashlimit-burst 10 --hashlimit-mode srcip --hashlimit-srcmask 24 --hashlimit-name limit_synproxy_ack -j log_limit_synproxy_ack # Ratelimit the ACK from 3WHS handled by SYNPROXY
iptables -t mangle -A PREROUTING -i enp3s0f0 -p tcp -m tcp --syn -m multiport --dports 2106,7777 -m hashlimit --hashlimit-above 5/sec --hashlimit-burst 10 --hashlimit-mode srcip --hashlimit-name limit_syn --hashlimit-srcmask 24 -j log_limit_syn # Limit SYN

### [INPUT] ###
iptables -A INPUT -i lo -j ACCEPT # Unlimited traffic on loopback
iptables -A INPUT -m set --match-set ipset_whitelist src -j ACCEPT # Whitelist
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT # Established connections
iptables -A INPUT -p icmp -m set --match-set ipset_allowed_ports src -m conntrack --ctstate NEW -j ACCEPT # Allowed ICMP
iptables -A INPUT -p tcp -m tcp -m multiport --dports 22,3306 -m set --match-set ipset_allowed_ports src -m conntrack --ctstate NEW -j ACCEPT # Allowed PORTS
iptables -A FORWARD -i enp3s0f0 -p tcp -m tcp -m multiport --dports 2106,7777 -m state --state INVALID,UNTRACKED -j SYNPROXY --sack-perm --timestamp --wscale 7 --mss 1460 # Catching state: UNTRACKED == SYN packets, INVALID   == ACK from 3WHS
iptables -A FORWARD -i enp3s0f0 -p tcp -m tcp -m multiport --dports 2106,7777 -m state --state INVALID -j DROP # Drop rest of state INVALID, this will e.g. catch SYN-ACK packet attacks

### [OUTPUT] ###
iptables -A OUTPUT -o lo -j ACCEPT # Unlimited traffic on loopback
iptables -A OUTPUT -m set --match-set ipset_whitelist src -j ACCEPT # Whitelist
iptables -A OUTPUT -m conntrack --ctstate ESTABLISHED -j ACCEPT # Established connections
iptables -A OUTPUT -p tcp -m tcp --tcp-flags ALL ACK,SYN -j ACCEPT # Allow the response to the SYN for the 3WHS before the connection is marked as established
iptables -A OUTPUT -p tcp -m tcp -m multiport --dports 80,443 -m conntrack --ctstate NEW -j ACCEPT # OS updates and Vote reward
#iptables -A OUTPUT -m limit --limit 1/second --limit-burst 5 -j LOG --log-prefix "output:drop: " --log-level 4 # Log output dropped packets;

Set default chain policies:
 

iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT DROP

 

Save All
 

sudo echo -n "" > /etc/ipset.conf # empty the ipset
ipset save > /etc/ipset.conf # save the ipset
service iptables save

We'll get more into how to capture/catch, monitor and handle all types of attacks in the next part; and to always have a Plan B.

Tips and Tricks

See conntrack used:
 

cat /proc/sys/net/netfilter/nf_conntrack_count

See conntrack max:
 

cat /proc/sys/net/netfilter/nf_conntrack_max

See SYNPROXY, if values are being updated, then it works:
 

watch -n1 cat /proc/net/stat/synproxy

Credits

Give me credits if you share it anywhere else, including my Discord and MxC topic's URL.
Discord: Trance#0694

Edited March 2, 2023 by Trance

[Trance]

P.S. I will have to remake all these guides at one point I'm not very happy with them but it should be enough for now.

[ERROR501]

You can do the same for your webhost if you have anything that needs to connect to your MySQL server; you could limit it to TCP and 3306 port though instead of accessing it all. --> If you have problem to connect your MySQL webhost to your MySQL game server, better turn off firewall and sniff your webhost IP Address before add your webhost IP Address to firewall.
As sometimes hosting use multiple IP Address for balancing and security reasons. TY @Trance

Edited May 4, 2023 by ERROR501