Guide The Linux Series! [PART 2]

Trance · January 27, 2023

Description

This topic is part of a multi-part series. We'll try to get everything straight to the point in this guide, without unnecessary over-explanation.

PART 1[CLICK HERE]

Work faster with a better terminal emulator.
Use a better editor.
Basic L2J server setup.
Manage and secure your MySQL server. [!]

PART 2 [THIS GUIDE]

Secure your Linux server.
Tuning system profiles. [!]
Network performance tuning. [!]
How to build and manage a firewall using iptables and conntrack - simplified version. [!]

PART 3 [CLICK HERE]

Understanding and managing the OVH Firewall. [!]
How to build and manage a firewall using iptables, conntrack, ipset and synproxy - advanced version. [!]
Mitigating most of the DDoS attacks. [!]

Secure Your Linux Server

Create a new user account. We should never log into our server as root:

adduser trance

Give your new user account sudo rights:

usermod -a -G sudo trance

Secure SSH Connection

We could use a SSH key instead SSH password authentication.

The keys are usually stored into user's directory/.ssh/authorized_keys
You can google how to generated a SSH key or you can simply use MobaXterm's:

Upload the SSH key:

ssh-copy-id trance@<ip_address>

WARNING - First double check if you can successfully login via the SSH key before disabling the SSH password authentication.
Now since we all have the same favorite editor nano, let's use it it to disable the SSH password authentication.

nano /etc/ssh/sshd_config

Change the following options to:

PasswordAuthentication no
PermitRootLogin no

Tuning System Profiles

Optimize the performance of a system by adjusting various device settings based on a variety of use case workloads.

Install and enable.

yum install tuned
systemctl enable --now tuned

You can see all available profiles with the following:

tuned-adm list

Available profiles:

balanced - Ideal for systems that require a compromise between power saving and performance.
desktop - Derived from the balanced profile. Provides faster response of interactive applications.
throughput-performance - Tunes the system for maximum throughput.
latency-performance - Ideal for server systems that require low latency at the expense of power consumption.
network-latency - Derived from the latency-performance profile. It enables additional network tuning parameters to provide low network latency.
network-throughput - Derived from the throughput-performance profile. Additional network tuning parameters are applied for maximum network throughput.
powersave - Tunes the system for maximum power saving.
oracle - Optimized for Oracle database loads based on the throughput-performance profile.
virtual-guest - Tunes the system for maximum performance if it runs on a virtual machine.
virtual-host - Tunes the system for maximum performance if it acts as a host for virtual machines.

See currently running profile:

tuned-adm active

I recommend latency-performance or network-latency.
You can choose a profile just like this:

tuned-adm recommend network-latency

Turn off tuned tuning activity with tuned-adm off.

tuned-adm off

Network Performance Tuning

We'll have to touch the HOT spot, the kernel. If you think an OS can handle it all by default, you're wrong!
We can view all your current kernel settings via:

sysctl -a

We can add our custom settings to be saved in the following config:

nano /etc/sysctl.conf

I've put all this together myself. You can google everything one by one if you'd like to know for what it is. I've added some useful comments. These values are well calculated, not randomly added.

# General
kernel.randomize_va_space = 0
net.core.netdev_max_backlog = 25000
net.core.rmem_max = 4136960
net.core.wmem_max = 4136960
net.ipv4.tcp_congestion_control = cubic
net.ipv4.tcp_fin_timeout = 1
net.ipv4.tcp_limit_output_bytes = 131072
net.ipv4.tcp_low_latency = 0
net.ipv4.tcp_max_tw_buckets = 45000
net.ipv4.tcp_rmem = 4096 87380 4136960
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_wmem = 4096 16384 4136960

# Desactivate the automatic conntrack helper assignment.
net.netfilter.nf_conntrack_helper = 0

# Enable the use of syncookies when the syn backlog queue is full.
net.ipv4.tcp_syncookies = 1

# SYNPROXY - This is necessary to have ACK packets (from 3WHS) marked as INVALID state.
# Disable picking up already established connections.
net.netfilter.nf_conntrack_tcp_loose = 0

# SYNPROXY - TCP timestamps as SYN cookies utilize this TCP option field.
# Tells the kernel to use timestamps as defined in RFC 1323.
net.ipv4.tcp_timestamps = 1

# Default size is calculated by dividing total memory
# by 16384 to determine the number of buckets but the hash table will
# never have fewer than 32 and limited to 16384 buckets. For systems
# with more than 4GB of memory it will be 65536 buckets.
net.netfilter.nf_conntrack_buckets = 500000

# SYNPROXY - it's recommended to do some conntrack entry tuning to increase the default 64K conn limit.
# nf_conntrack_buckets * 4
net.netfilter.nf_conntrack_max = 2000000

# nf_conntrack_max / 4
# as reference only: echo 500000 > /sys/module/nf_conntrack/parameters/hashsize

net.netfilter.nf_conntrack_tcp_timeout_established = 1800 # default 432000 (5 days); 1800 = 30 minutes
net.netfilter.nf_conntrack_tcp_timeout_close = 10 # default: 10
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 20 # default: 60
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 30 # default: 120
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 30 # default: 30
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 30 # default: 60
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 30 # default: 120
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 20 # default: 120

And then...

Applying all of the above:

sysctl -p

Running the following as well have it all set:

echo 2000000 > /sys/module/nf_conntrack/parameters/hashsize
echo 0 > /proc/sys/net/netfilter/nf_conntrack_helper

Our conntrack can handle so many connections now!

IPTables and Conntrack

Conntrack allows us to use NEW, ESTABLISHED, RELATED states in ipables for the incoming connections. So we need to store all that info!
We need to disable and replace the firewalld with iptables:

sudo yum remove firewalld -y
sudo yum install iptables-services -y
sudo yum install conntrack-tools -y
sudo yum install ipset -y
sudo yum install ipset-service -y
sudo systemctl start iptables
sudo systemctl start ip6tables
sudo systemctl enable iptables
sudo systemctl enable ip6tables

The following commands will be used to flush the entire iptables every time we f*ck it up.

iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -t raw -F
iptables -t raw -X
iptables -F
iptables -X
iptables -t filter -P INPUT ACCEPT
iptables -t filter -P FORWARD ACCEPT
iptables -t filter -P OUTPUT ACCEPT
iptables -t filter -F
iptables -t filter -X
service iptables save

IPTables' Chains

Chains - A chain is a string of rules. When a packet is received, iptables finds the appropriate table, then runs it through the chain of rules until it finds a match.

Rules - A rule is a statement that tells the system what to do with a packet. Rules can block one type of packet, or forward another type of packet.

To make it easier for us to understand how the chains in iptables work:

You’ll find that most if not all guides on how to block DDoS attacks using iptables use the filter table and the INPUT chain for anti-DDoS rules. The issue with this approach is that the INPUT chain is only processed after the PREROUTING and FORWARD chains and therefore only applies if the packet doesn’t match any of these two chains. This causes a delay in the filtering of the packet which consumes resources. In conclusion, to make our rules as effective as possible, we need to move our anti-DDoS rules as far up the chains as possible. The first chain that can apply to a packet is the PREROUTING chain, so ideally we’ll want to filter the bad packets in this chain already.

The Actual Basic Rules

We can use the following command to see all rules in a particular table:

iptables -t <table_name> -L

Like iptables -t mangle -L or simply iptables -L for the default table a.k.a. the FILTER table.

MANGLE Table

PREROUTING

Like it was described above, we need to save resources. So we'll have the following basic rules up in the mangle table.

1. Drop INVALID packets, which means the incoming connection is neither NEW, RELATED, or ESTABLISHED. Not needed on Part 3 when SYNPROXY is being used.

iptables -t mangle -A PREROUTING -m conntrack --ctstate INVALID -j DROP

2. Drop the TCP packet if it's NEW and NOT a SYN. A TCP 3-way handshake should always start with a SYN. So the attacker can't exploit that.

iptables -t mangle -A PREROUTING -p tcp ! --syn -m conntrack --ctstate NEW -j DROP

3. Drop SYN packets with weird MSS value

iptables -t mangle -A PREROUTING -p tcp -m conntrack --ctstate NEW -m tcpmss ! --mss 536:65535 -j DROP

4. Block packets with bogus TCP flags - basically different set of unusual flags.

iptables -t mangle -A PREROUTING -p tcp --tcp-flags ACK,FIN FIN -j DROP
iptables -t mangle -A PREROUTING -p tcp --tcp-flags ACK,PSH PSH -j DROP
iptables -t mangle -A PREROUTING -p tcp --tcp-flags ACK,URG URG -j DROP
iptables -t mangle -A PREROUTING -p tcp --tcp-flags FIN,RST FIN,RST -j DROP
iptables -t mangle -A PREROUTING -p tcp --tcp-flags SYN,FIN SYN,FIN -j DROP
iptables -t mangle -A PREROUTING -p tcp --tcp-flags SYN,RST SYN,RST -j DROP
iptables -t mangle -A PREROUTING -p tcp --tcp-flags ALL ALL -j DROP
iptables -t mangle -A PREROUTING -p tcp --tcp-flags ALL NONE -j DROP
iptables -t mangle -A PREROUTING -p tcp --tcp-flags ALL FIN,PSH,URG -j DROP
iptables -t mangle -A PREROUTING -p tcp --tcp-flags ALL SYN,FIN,PSH,URG -j DROP
iptables -t mangle -A PREROUTING -p tcp --tcp-flags ALL SYN,RST,ACK,FIN,URG -j DROP

5. Block spoofed packets - no explanation is needed I guess.

iptables -t mangle -A PREROUTING -s 224.0.0.0/3 -j DROP
iptables -t mangle -A PREROUTING -s 169.254.0.0/16 -j DROP
iptables -t mangle -A PREROUTING -s 172.16.0.0/12 -j DROP
iptables -t mangle -A PREROUTING -s 192.0.2.0/24 -j DROP
iptables -t mangle -A PREROUTING -s 192.168.0.0/16 -j DROP
iptables -t mangle -A PREROUTING -s 10.0.0.0/8 -j DROP
iptables -t mangle -A PREROUTING -s 0.0.0.0/8 -j DROP
iptables -t mangle -A PREROUTING -s 240.0.0.0/5 -j DROP
iptables -t mangle -A PREROUTING -s 127.0.0.0/8 ! -i lo -j DROP

FILTER Table

The next table we'll use is the one most people use. Like its name, we filter stuff in here.

INPUT - incoming connections

1. Unlimited traffic on (local) loopback

iptables -A INPUT -i lo -j ACCEPT

2. We no longer need to filter connections that are already ESTABLISHED.

iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

3. Allowing OVH's Gateway, DNS and NTP servers. So we avoid clock sync issues and such.

iptables -A INPUT -s 198.244.200.254 -j ACCEPT
iptables -A INPUT -s cdns.ovh.net -j ACCEPT
iptables -A INPUT -s ntp0.ovh.net -j ACCEPT

4A. Next step we need to allow access to ourselves via SSH. Assuming my IP is 51.10.10.10.

iptables -A INPUT -p tcp -m tcp -s 51.10.10.10 --dport 22 -m conntrack --ctstate NEW -j ACCEPT

4B. If my Home IP is changing dynamically, most of the time it stays in the same CIDR - 51.10.10.0/24 from 51.10.10.0 to 51.10.10.255 or 51.10.0.0/16 from 51.10.0.0 to 51.10.255.255.

iptables -A INPUT -p tcp -m tcp -s 51.10.10.0/24 --dport 22 -m conntrack --ctstate NEW -j ACCEPT

4C. The best option is to have a VPN with a Dedicated IP (or a static IP at Home) and to allow that IP to access it all.

iptables -A INPUT -p tcp -m tcp -s 51.10.10.10/32 -m conntrack --ctstate NEW -j ACCEPT

5A. Allow everyone to access the Login and Game servers on their specific ports.

iptables -A INPUT -p tcp -m tcp --dport 2106 -m conntrack --ctstate NEW -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 7777 -m conntrack --ctstate NEW -j ACCEPT

5B. We can add multiple ports in a rule though.

iptables -A INPUT -p tcp -m tcp -m multiport --dports 2106,7777 -m conntrack --ctstate NEW -j ACCEPT

OUTPUT - from the machine outside

1. Unlimited traffic on (local) loopback

iptables -A OUTPUT -o lo -j ACCEPT

2. We no longer need to filter connections that are already ESTABLISHED.

iptables -A OUTPUT -m conntrack --ctstate ESTABLISHED -j ACCEPT

3. Allow the response to the SYN for the 3-way handshake before the connection is marked as ESTABLISHED.

iptables -A OUTPUT -p tcp -m tcp --tcp-flags ALL ACK,SYN -j ACCEPT

3. Allowing OVH's Gateway, DNS and NTP servers. So we avoid clock sync issues and such.

iptables -A OUTPUT -d 198.244.200.254 -j ACCEPT
iptables -A OUTPUT -d cdns.ovh.net -j ACCEPT
iptables -A OUTPUT -d ntp0.ovh.net -j ACCEPT

4. Allow OS updates and Vote reward.

iptables -A OUTPUT -p tcp -m tcp -m multiport --dports 80,443 -m conntrack --ctstate NEW -j ACCEPT

5. Log output dropped packets; it's good for debugging to see why our server can't access something. You'll see it in the kernel logs. You can do the same for INPUT if you'd like to see what you do wrong - be aware of the spam!

iptables -A OUTPUT -m limit --limit 1/second --limit-burst 5 -j LOG --log-prefix "output:drop: " --log-level 4

You can see kernel logs the same way you'd watch the Game Server console.

tail -f /var/log/messages

Set default chain policies:

iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT DROP

Tips and Tricks

See the rules and their numbers on the mangle table:

iptables -t mangle -L --line-number

See the rules and their numbers on the filter table:

iptables -t -L --line-number

Delete a rule using the line number on the mangle table:

iptables -t mangle -D PREROUTING 69

Delete a rule using the line number on the filter table:

iptables -D INPUT 69

We'll get into more advanced and complicated firewall practices in the next part!

Save All

I'm pretty sure this is different on other distros. You may even create a list and always store all your rules in there, and then add that to the startup.