Guide HOWTO - L2 server with (transparent) proxies

[eressea]

HOWTO - L2 server with (transparent) proxies

DISCLAIMER: I wrote this HOWTO without really testing it (I've configured it for my server more than year ago), so if you find any mistake, feel free to comment/ask, I'll try to update this HOWTO appropriately.

Purpose of using proxies

The standard old way of setting up L2off server involves one Windows Server that is connected directly to internet and players connect directly to it's IP address. While it's enough for a small server, it may bring few problems:

• If someone wants to DDoS your server, he just pays for some attack to your IP address. You're screwed.
• There may be players who experience lags because route from their network to your server isn't perfect (because of bad peering between player's ISP and your datacenter etc). You can't just tell those players "Try better ISP".
• Players from foreign countries may have good ping to datacenters in their country but not to your data center.

Some of these problems can be mitigated by using proxies for your server:

• When someone buys DDoS attack to your login server on first proxy, you can just change IP address in your DNS and let your players login on your second proxy (you need to have domain name instead of IP address in your l2.ini).
• When someone buys DDoS attack to your game server on first proxy, players can still play on second proxy.
• When player has bad ping on first proxy, login will be bit slower but player can select second proxy with better ping and play on it

You can also do the same with your website, just replace ports 2106 and 7777 with 80 and 443...

What will you need

• Some basic knowledge about L2off, Linux and networking. Also you'll have to read few manuals.
• Linux router between your L2 server and internet, you can do it two ways (of course it must be configured to forward IPv4 traffic):
  • real physical Linux router between them
  • Linux as main operating system on your server hardware + L2 server in virtual machine (kvm-qemu).
• Linux servers (1 core & 512 MB RAM is usually enough) for proxies
• Copy of patched hauthd - READ INSTRUCTIONS THERE
• Script that will be able to restart hauthd when L2 server crashes/exits

How to do it

I will use following IP networks and addresses in this example:

• Networks:
  • Local network 10.0.0.0/24 between Linux router and Windows VM
  • Virtual network 10.0.1.0/24 between Linux router and proxies
• Addresses:
  • 1.2.3.4 on Linux router eth0
  • 2.3.4.5 on first proxy eth0
  • 3.4.5.6 on second proxy eth0
  • 4.5.6.7 on third proxy eth0
  • 10.0.0.1 on Linux router eth1 (for physical router) or tap0 (for kvm-qemu variant)
  • 10.0.0.2 on Windows server Local network interface
  • 10.0.1.1 on Linux router tun0 (OpenVPN server 1)
  • 10.0.2.1 on Linux router tun1 (OpenVPN server 2)
  • 10.0.3.1 on Linux router tun2 (OpenVPN server 3)
  • 10.0.1.2 on first proxy tun0 (OpenVPN client)
  • 10.0.2.2 on second proxy tun0 (OpenVPN client)
  • 10.0.3.2 on third proxy tun0 (OpenVPN client)

 

Physical network:

 

 

Logical network:

 

 

Now proceed with installation and configuration:

• ROUTER + PROXIES: Install OpenVPN
• ROUTER: For each proxy, configure one instance of OpenVPN server with static key (it will have to run on different ports, for example 1194, 1195, 1196)
• PROXY: Copy static key from router, configure OpenVPN client
• WINDOWS: Make sure windows server can ping linux router (ping 10.0.0.1)
• PROXY: Make sure proxies can ping server (ping 10.0.1.1 / ping 10.0.2.1 / ping 10.0.3.1)
• PROXY: Setup NAT (find some way how to run it on every start!)

  iptables -t nat -A PREROUTING -m tcp -p tcp --dport 2106 -j DNAT --to-destination 10.0.0.2:2106
  iptables -t nat -A PREROUTING -m tcp -p tcp --dport 7777 -j DNAT --to-destination 10.0.0.2:7777
  

  You need this to make proxies forward traffic on ports 2106 and 7777 to windows server.
   
• ROUTER: Setup connmark (find some way how to run it on every start! If it's physical router, use eth1 instead of tap0!):

  iptables -t mangle -A PREROUTING -i tun0 -p tcp -m tcp --dport 7777 -j CONNMARK --set-mark 100
  iptables -t mangle -A PREROUTING -i tun1 -p tcp -m tcp --dport 2106 -j CONNMARK --set-mark 101
  iptables -t mangle -A PREROUTING -i tun1 -p tcp -m tcp --dport 7777 -j CONNMARK --set-mark 101
  iptables -t mangle -A PREROUTING -i tun2 -p tcp -m tcp --dport 2106 -j CONNMARK --set-mark 101
  iptables -t mangle -A PREROUTING -i tun2 -p tcp -m tcp --dport 7777 -j CONNMARK --set-mark 101
  iptables -t mangle -i tap0 -p tcp -m tcp --sport 2106 -j CONNMARK --restore-mark
  iptables -t mangle -i tap0 -p tcp -m tcp --sport 7777 -j CONNMARK --restore-mark

  You need this to mark connections so you know where they came from.
   
• ROUTER: Setup policy-based routing (find some way how to run it on every start!):

  ip rule add fwmark 100 table 100
  ip route add default via 10.0.1.2 table 100
  ip rule add fwmark 101 table 101
  ip route add default via 10.0.2.2 table 101
  ip rule add fwmark 102 table 102
  ip route add default via 10.0.3.2 table 102

  You need this to make packets go back through the right proxy. Otherwise you'll send those packets out via your router public interface which won't work.
   

• WINDOWS: Setup hauthd to use multiple proxies and add servers appropriately to dbo.server table
• WINDOWS: Make sure hauthd restarts every time L2 server restarts

 

Useful resources

• Patched hauthd with basic instructions
• OpenVPN Static Key Mini-HOWTO
• Linux Advanced Routing & Traffic Control HOWTO - Rules - routing policy database
• Linux Advanced Routing & Traffic Control HOWTO - Netfilter & iproute - marking packets

 

 

Edited December 5, 2017 by eressea

[Erinia]

+ 1 amazing share from eressea

 

 

[bit]

Would this work with single proxy and no router machine? I am kind of banging my head here trying to make that setup work.

 

My setup: 1 linux proxy running ppp server.

Proxy ppp0 interface with ip 192.168.0.1, windows server client with ip 192.168.0.234.

$ip in the commands is external proxy server ip.

 

iptables -t nat    -A PREROUTING         -p tcp -m multiport --dport 2106,7777        -j DNAT     --to-dest 192.168.0.234
iptables -t mangle -A PREROUTING -i eth0 -p tcp -m multiport --dport 2106,7777 -m tcp -j CONNMARK --set-mark 101
iptables -t mangle -A PREROUTING -i ppp0 -p tcp -m multiport --sport 2106,7777 -m tcp -j CONNMARK --restore-mark
ip rule add fwmark 101 table 101
ip route add default via $ip table 101

Any idea what i do wrong here?

 

This works, but server sees proxy IP as player IP:

iptables -t nat -A PREROUTING  -p tcp -m multiport --dport 2106,7777 -j DNAT       --to-dest 192.168.0.234
iptables -t nat -A POSTROUTING                                       -j MASQUERADE

 

By the way first post has some typos. Chain is missing in commands with "--restore-mark"

[eressea]

52 minutes ago, bit said:

Would this work with single proxy and no router machine? I am kind of banging my head here trying to make that setup work.

 

My setup: 1 linux proxy running ppp server.

Proxy ppp0 interface with ip 192.168.0.1, windows server client with ip 192.168.0.234.

$ip in the commands is external proxy server ip.

 

iptables -t nat    -A PREROUTING         -p tcp -m multiport --dport 2106,7777        -j DNAT     --to-dest 192.168.0.234
iptables -t mangle -A PREROUTING -i eth0 -p tcp -m multiport --dport 2106,7777 -m tcp -j CONNMARK --set-mark 101
iptables -t mangle -A PREROUTING -i ppp0 -p tcp -m multiport --sport 2106,7777 -m tcp -j CONNMARK --restore-mark
ip rule add fwmark 101 table 101
ip route add default via $ip table 101

Any idea what i do wrong here?

 

This works, but server sees proxy IP as player IP:

iptables -t nat -A PREROUTING  -p tcp -m multiport --dport 2106,7777 -j DNAT       --to-dest 192.168.0.234
iptables -t nat -A POSTROUTING                                       -j MASQUERADE

 

By the way first post has some typos. Chain is missing in commands with "--restore-mark"

 

You can't do it without Linux router - Windows can't do policy-based routing.

 

You would have to set default gateway on Windows Server to 192.168.0.1 and make all packets out go through it (but you would lose connectivity to it from internet, so you would have to create static route via previous default gateway to your proxy; even that would be very inconvenient because you would have to access your server only from proxy or via the proxy - because there would be no other way to it).

 

Without right default gateway your only option is to use MASQUERADE or SNAT (it's the same, masquerade just automatically assumes new source IP address) which changes sender IP address to proxy IP - that you then see on Windows server.

Edited December 5, 2017 by eressea

[tiagopyerri]

4 hours ago, eressea said:

HOWTO - L2 server with (transparent) proxies

DISCLAIMER: I wrote this HOWTO without really testing it (I've configured it for my server more than year ago), so if you find any mistake, feel free to comment/ask, I'll try to update this HOWTO appropriately.

Purpose of using proxies

The standard old way of setting up L2off server involves one Windows Server that is connected directly to internet and players connect directly to it's IP address. While it's enough for a small server, it may bring few problems:

• If someone wants to DDoS your server, he just pays for some attack to your IP address. You're screwed.
• There may be players who experience lags because route from their network to your server isn't perfect (because of bad peering between player's ISP and your datacenter etc). You can't just tell those players "Try better ISP".
• Players from foreign countries may have good ping to datacenters in their country but not to your data center.

Some of these problems can be mitigated by using proxies for your server:

• When someone buys DDoS attack to your login server on first proxy, you can just change IP address in your DNS and let your players login on your second proxy (you need to have domain name instead of IP address in your l2.ini).
• When someone buys DDoS attack to your game server on first proxy, players can still play on second proxy.
• When player has bad ping on first proxy, login will be bit slower but player can select second proxy with better ping and play on it

You can also do the same with your website, just replace ports 2106 and 7777 with 80 and 443...

What will you need

• Some basic knowledge about L2off, Linux and networking. Also you'll have to read few manuals.
• Linux router between your L2 server and internet, you can do it two ways (of course it must be configured to forward IPv4 traffic):
  • real physical Linux router between them
  • Linux as main operating system on your server hardware + L2 server in virtual machine (kvm-qemu).
• Linux servers (1 core & 512 MB RAM is usually enough) for proxies
• Copy of patched hauthd - READ INSTRUCTIONS THERE
• Script that will be able to restart hauthd when L2 server crashes/exits

How to do it

I will use following IP networks and addresses in this example:

• Networks:
  • Local network 10.0.0.0/24 between Linux router and Windows VM
  • Virtual network 10.0.1.0/24 between Linux router and proxies
• Addresses:
  • 1.2.3.4 on Linux router eth0
  • 2.3.4.5 on first proxy eth0
  • 3.4.5.6 on second proxy eth0
  • 4.5.6.7 on third proxy eth0
  • 10.0.0.1 on Linux router eth1 (for physical router) or tap0 (for kvm-qemu variant)
  • 10.0.0.2 on Windows server Local network interface
  • 10.0.1.1 on Linux router tun0 (OpenVPN server 1)
  • 10.0.2.1 on Linux router tun1 (OpenVPN server 2)
  • 10.0.3.1 on Linux router tun2 (OpenVPN server 3)
  • 10.0.1.2 on first proxy tun0 (OpenVPN client)
  • 10.0.2.2 on second proxy tun0 (OpenVPN client)
  • 10.0.3.2 on third proxy tun0 (OpenVPN client)

 

Physical network:

 

 

Logical network:

 

 

Now proceed with installation and configuration:

• ROUTER + PROXIES: Install OpenVPN
• ROUTER: For each proxy, configure one instance of OpenVPN server with static key (it will have to run on different ports, for example 1194, 1195, 1196)
• PROXY: Copy static key from router, configure OpenVPN client
• WINDOWS: Make sure windows server can ping linux router (ping 10.0.0.1)
• PROXY: Make sure proxies can ping server (ping 10.0.1.1 / ping 10.0.2.1 / ping 10.0.3.1)
• PROXY: Setup NAT (find some way how to run it on every start!)

  
  iptables -t nat -A PREROUTING -m tcp -p tcp --dport 2106 -j DNAT --to-destination 10.0.0.2:2106
  iptables -t nat -A PREROUTING -m tcp -p tcp --dport 7777 -j DNAT --to-destination 10.0.0.2:7777
  

  You need this to make proxies forward traffic on ports 2106 and 7777 to windows server.
   
• ROUTER: Setup connmark (find some way how to run it on every start! If it's physical router, use eth1 instead of tap0!):

  
  iptables -t mangle -A PREROUTING -i tun0 -p tcp -m tcp --dport 7777 -j CONNMARK --set-mark 100
  iptables -t mangle -A PREROUTING -i tun1 -p tcp -m tcp --dport 2106 -j CONNMARK --set-mark 101
  iptables -t mangle -A PREROUTING -i tun1 -p tcp -m tcp --dport 7777 -j CONNMARK --set-mark 101
  iptables -t mangle -A PREROUTING -i tun2 -p tcp -m tcp --dport 2106 -j CONNMARK --set-mark 101
  iptables -t mangle -A PREROUTING -i tun2 -p tcp -m tcp --dport 7777 -j CONNMARK --set-mark 101
  iptables -t mangle -i tap0 -p tcp -m tcp --sport 2106 -j CONNMARK --restore-mark
  iptables -t mangle -i tap0 -p tcp -m tcp --sport 7777 -j CONNMARK --restore-mark

  You need this to mark connections so you know where they came from.
   
• ROUTER: Setup policy-based routing (find some way how to run it on every start!):

  
  ip rule add fwmark 100 table 100
  ip route add default via 10.0.1.2 table 100
  ip rule add fwmark 101 table 101
  ip route add default via 10.0.2.2 table 101
  ip rule add fwmark 102 table 102
  ip route add default via 10.0.3.2 table 102

  You need this to make packets go back through the right proxy. Otherwise you'll send those packets out via your router public interface which won't work.
   

• WINDOWS: Setup hauthd to use multiple proxies and add servers appropriately to dbo.server table
• WINDOWS: Make sure hauthd restarts every time L2 server restarts

 

Useful resources

• Patched hauthd with basic instructions
• OpenVPN Static Key Mini-HOWTO
• Linux Advanced Routing & Traffic Control HOWTO - Rules - routing policy database
• Linux Advanced Routing & Traffic Control HOWTO - Netfilter & iproute - marking packets

 

 

and it's nice to have people who can teach others how to have a professional server these days, where servers do not exceed 90 days
 Deserves to be Fixed this topic

[bit]

Actually i figured it out. Take a look.

 

Linux proxy server runs OpenVPN server.

OpenVPN network IP of proxy: 10.200.0.1

Windows connects to OpenVPN server, gets IP 10.200.0.10

AuthD/Server both listen on address 10.200.0.10

Do not set default gateway on windows server to 10.200.0.1 This is irrelevant. OpenVPN client does it automatically.

$ip as always - external IP of proxy server

Proxy firewall setup:

iptables        -A INPUT                 -s 10.200.0.0/24 -d 10.200.0.1                         -j ACCEPT
iptables -t nat -A POSTROUTING -o eth0   -s 10.200.0.0/24 -d 10.200.0.10                        -j SNAT --to $ip
iptables -t nat -A PREROUTING  -i eth0 ! -s 10.200.0.10   -p tcp -m multiport --dport 2106,7777 -j DNAT --to 10.200.0.10

To my surprise this setup server can see real client IP address. Also if default gateway is not set on windows everything still works and windows server can still access internet.

 

Can not try it with ppp yet because live server, but i suspect there would be no difference (except for lan IP addresses ofc).

 

Edit:

Turns out windows server indeed can not access internet. Still trying to figure out how to fix openvpn routes so internet works as well as proxied server.

 

Edit:

K i finally figured out how to make internet connection and forwarded server work as well. This is rather dirty solution but it works.

 

Key requirement is that you set default gateway to OpenVPN adapter on windows server. For some reason it is not done automatically.

Add following settings to your OpenVPN.ovpn client config file:

route-nopull
script-security 2
up-delay
up "setgw.bat"

Remove "redirect-gateway" option if you have it.

 

Create setgw.bat script in openvpn config folder with content:

start cmd /c "timeout 10 && netsh interface ipv4 set address %1 static %4 255.255.255.0 %5"

This file will sleep a bit and set default gateway to openvpn network adapter.

 

If you know better solution please share, because this one sucks.

Edited December 5, 2017 by bit

[eressea]

17 hours ago, bit said:

If you know better solution please share, because this one sucks.

 

How about

• install VirtualBox (or use Hyper-V but I don't have any experience with it)
• create virtual machine with 1 core, 512 MB RAM and 10 GB disk image
• install Debian Stretch with just only ssh server
• configure networking so
  • host (windows) will have no IP address set on the primary NIC
  • guest (linux) will have eth0 bridged with windows primary NIC and will have IP address that your windows server had
  • guest (linux) will have eth1 just between host and guest and set some internal address on it (10.0.0.1/24)
  • host (windows) virtual NIC (linux eth1) will have internal address in the same segment (10.0.0.2/24) and default gateway set to linux IP (10.0.0.1)

Then you would have real linux router so you can do all the stuff needed (policy-based routing). But you need to find some way how to start that VM when windows start.

[tk422]

This is a great post. Thanks for taking the time to write it!

 

I would like to offer a few improvements as outlined below. This system i propose borrows heavily from my career life and experience. I've deployed and operate several of these systems and use one of my own to host professional business applications.

 

 

I intentionally did not provide step by step instructions on how to set this up. It is important that you actually spend the time to understand how the system works rather than blindly follow instructions. Nothing presented here is overly complex but it will likely require you to learn something new.

 

/// Resources

Online Diagram: L2 Server Hosting Environment

 

OVH Virtual Mac (Other providers offer this too!)
https://docs.ovh.com/gb/en/dedicated/network-virtual-mac/

 

MikroTik CHR
https://wiki.mikrotik.com/wiki/Manual:CHR

 

 

 

Edited December 10, 2017 by tk422

[bit]

eressea trying to replicate your exact setup. im banging my head here. sure you did not forget anything about proxy setup? you set DNAT to IP of l2server, however here on my end proxy does not know where to route such packet because l2server is on different subnet than openvpn proxies. any ideas?

[eressea]

11 hours ago, bit said:

eressea trying to replicate your exact setup. im banging my head here. sure you did not forget anything about proxy setup? you set DNAT to IP of l2server, however here on my end proxy does not know where to route such packet because l2server is on different subnet than openvpn proxies. any ideas?

 

Hi, the machine "Linux Router" must be in all networks and route packets between them

[bit]

1 hour ago, eressea said:

 

Hi, the machine "Linux Router" must be in all networks and route packets between them

Ah right, when proxy connects to router it gets default gateway set. Well truth is im not doing 100% replica of this setup. I swapped openvpn with wireguard. Things work differently with wireguard. Now it all makes sense. Thanks!

[mdavid]

Hi

could anyone please give me a hand in this? (not free)

I can't get it done

[bit]

I finally managed to replicate your setup almost exactly. The only difference is that i used wireguard instead of openvpn. The most notable thing in this setup is rp_filter. I was not aware of it and thus spent multiple weeks figuring it out. Anyone following these directions:

• Do not customize AllowedIPs thinking you will tighten security - you will break packet routing.
• Do not forget rp_filter part.
• You do not have to run windows instance in a VM. Could simply connect windows server to same network using TunSafe client.

eressea thanks for initial guide. it was absolutely essential in making transparent proxying happen for me :)

#### Setup

+--+-------------------+------+              +----------------------------+
|  |                   |      |              |                            |
|  | VM (10.20.0.100)  <---------------------+  Proxy: 10.21.0.101 (wg21) |
|  |                   |      |              |         y.y.y.y     (eth0) |
|  +-------------------+      |              |                            |
|                             |              +----------------------------+
|  Router: 10.20.0.1 (virbr0) |
|          10.21.0.1 (wg21)   |
|          x.x.x.x   (eth0)   |
|                             |
+-----------------------------+

#### Server

 

wg21.conf

    [Interface]
    Address = 10.21.0.1/24
    Table = off
    SaveConfig = true
    ListenPort = 43832
    PrivateKey = <...>
    
    [Peer]
    PublicKey = <...>
    AllowedIPs = 0.0.0.0/0        # Required for preserving original source ip
    Endpoint = y.y.y.y:43832

iptables

    # enable forwarding
    echo 1 > /proc/sys/net/ipv4/ip_forward
    # prevent packets with original source from being dropped
    echo 0 >/proc/sys/net/ipv4/conf/all/rp_filter
    echo 1 >/proc/sys/net/ipv4/conf/eth0/rp_filter
    
    wg-quick up wg21
    
    # transparent proxying
    ip rule add fwmark 101 table 101
    ip route add default via 10.21.0.101 table 101
    iptables -t mangle -A PREROUTING -i wg21   -p tcp -m multiport --dport $gs_ports -j CONNMARK --set-mark 101
    iptables -t mangle -A PREROUTING -i virbr0 -p tcp -m multiport --sport $gs_ports -j CONNMARK --restore-mark
    
    # forwarding
    iptables -t filter -I FORWARD -i wg21 -o virbr0 -p tcp -d $gs -m multiport --dports $gs_ports -j ACCEPT
    iptables -t filter -A FORWARD -i wg21 -j ACCEPT
    iptables -t filter -A FORWARD -o wg21 -j ACCEPT

#### Proxy

wg21.conf

    [Interface]
    Address = 10.21.0.101/32
    Table = off
    SaveConfig = true
    ListenPort = 43832
    PrivateKey = <...>
    
    [Peer]
    PublicKey = <...>
    AllowedIPs = 0.0.0.0/0    # Required so that packets can reach 10.20.0.100
    Endpoint = x.x.x.x:43832

iptables

    wg-quick up wg21
    ip route add 10.20.0.100 dev wg21    # Routing to VM
    ip route add 10.21.0.0/24 dev wg21   # Routing to vlan
    
    iptables -A FORWARD -i wg21 -j ACCEPT
    iptables -A FORWARD -o wg21 -j ACCEPT
    
    # forward incoming packets to VM
    iptables -t nat -A PREROUTING  -i $eth0 -p tcp -m multiport --dport $gs_ports -j DNAT --to 10.20.0.100
    iptables -t nat -A POSTROUTING -s 10.21.0.0/24 -o eth0 -j MASQUERADE

 

[eressea]

2 hours ago, bit said:

I finally managed to replicate your setup almost exactly.

 

Nice :)

[Hitcher]

Tell me please, is it possible to set up account registration with binding to static ip, and adding ip to the white list - with the included server login function only from the white list? And enable an additional firewall, which will also filter out all ip addresses except the white list? Will this protect against possible DDoS attacks?