eressea

Guide HOWTO - L2 server with (transparent) proxies

8 posts in this topic

eressea    12

HOWTO - L2 server with (transparent) proxies

DISCLAIMER: I wrote this HOWTO without really testing it (I've configured it for my server more than year ago), so if you find any mistake, feel free to comment/ask, I'll try to update this HOWTO appropriately.

Purpose of using proxies

The standard old way of setting up L2off server involves one Windows Server that is connected directly to internet and players connect directly to it's IP address. While it's enough for a small server, it may bring few problems:

  • If someone wants to DDoS your server, he just pays for some attack to your IP address. You're screwed.
  • There may be players who experience lags because route from their network to your server isn't perfect (because of bad peering between player's ISP and your datacenter etc). You can't just tell those players "Try better ISP".
  • Players from foreign countries may have good ping to datacenters in their country but not to your data center.

Some of these problems can be mitigated by using proxies for your server:

  • When someone buys DDoS attack to your login server on first proxy, you can just change IP address in your DNS and let your players login on your second proxy (you need to have domain name instead of IP address in your l2.ini).
  • When someone buys DDoS attack to your game server on first proxy, players can still play on second proxy.
  • When player has bad ping on first proxy, login will be bit slower but player can select second proxy with better ping and play on it

You can also do the same with your website, just replace ports 2106 and 7777 with 80 and 443...

What will you need

  • Some basic knowledge about L2off, Linux and networking. Also you'll have to read few manuals.
  • Linux router between your L2 server and internet, you can do it two ways (of course it must be configured to forward IPv4 traffic):
    • real physical Linux router between them
    • Linux as main operating system on your server hardware + L2 server in virtual machine (kvm-qemu).
  • Linux servers (1 core & 512 MB RAM is usually enough) for proxies
  • Copy of patched hauthd - READ INSTRUCTIONS THERE
  • Script that will be able to restart hauthd when L2 server crashes/exits

How to do it

I will use following IP networks and addresses in this example:

  • Networks:
    • Local network 10.0.0.0/24 between Linux router and Windows VM
    • Virtual network 10.0.1.0/24 between Linux router and proxies
  • Addresses:
    • 1.2.3.4 on Linux router eth0
    • 2.3.4.5 on first proxy eth0
    • 3.4.5.6 on second proxy eth0
    • 4.5.6.7 on third proxy eth0
    • 10.0.0.1 on Linux router eth1 (for physical router) or tap0 (for kvm-qemu variant)
    • 10.0.0.2 on Windows server Local network interface
    • 10.0.1.1 on Linux router tun0 (OpenVPN server 1)
    • 10.0.2.1 on Linux router tun1 (OpenVPN server 2)
    • 10.0.3.1 on Linux router tun2 (OpenVPN server 3)
    • 10.0.1.2 on first proxy tun0 (OpenVPN client)
    • 10.0.2.2 on second proxy tun0 (OpenVPN client)
    • 10.0.3.2 on third proxy tun0 (OpenVPN client)

 

Physical network:

 

l2-proxies-physical-network.png.aced8219e6456f630529db2754136d09.png

 

Logical network:

 

l2-proxies-logical-network.png.8dc15fb59fab157a254514dddca94364.png

 

Now proceed with installation and configuration:

  • ROUTER + PROXIES: Install OpenVPN
  • ROUTER: For each proxy, configure one instance of OpenVPN server with static key (it will have to run on different ports, for example 1194, 1195, 1196)
  • PROXY: Copy static key from router, configure OpenVPN client
  • WINDOWS: Make sure windows server can ping linux router (ping 10.0.0.1)
  • PROXY: Make sure proxies can ping server (ping 10.0.1.1 / ping 10.0.2.1 / ping 10.0.3.1)
  • PROXY: Setup NAT (find some way how to run it on every start!)
    iptables -t nat -A PREROUTING -m tcp -p tcp --dport 2106 -j DNAT --to-destination 10.0.0.2:2106
    iptables -t nat -A PREROUTING -m tcp -p tcp --dport 7777 -j DNAT --to-destination 10.0.0.2:7777
    
    You need this to make proxies forward traffic on ports 2106 and 7777 to windows server.
     
  • ROUTER: Setup connmark (find some way how to run it on every start! If it's physical router, use eth1 instead of tap0!):
    iptables -t mangle -A PREROUTING -i tun0 -p tcp -m tcp --dport 7777 -j CONNMARK --set-mark 100
    iptables -t mangle -A PREROUTING -i tun1 -p tcp -m tcp --dport 2106 -j CONNMARK --set-mark 101
    iptables -t mangle -A PREROUTING -i tun1 -p tcp -m tcp --dport 7777 -j CONNMARK --set-mark 101
    iptables -t mangle -A PREROUTING -i tun2 -p tcp -m tcp --dport 2106 -j CONNMARK --set-mark 101
    iptables -t mangle -A PREROUTING -i tun2 -p tcp -m tcp --dport 7777 -j CONNMARK --set-mark 101
    iptables -t mangle -i tap0 -p tcp -m tcp --sport 2106 -j CONNMARK --restore-mark
    iptables -t mangle -i tap0 -p tcp -m tcp --sport 7777 -j CONNMARK --restore-mark
    You need this to mark connections so you know where they came from.
     
  • ROUTER: Setup policy-based routing (find some way how to run it on every start!):
    ip rule add fwmark 100 table 100
    ip route add default via 10.0.1.2 table 100
    ip rule add fwmark 101 table 101
    ip route add default via 10.0.2.2 table 101
    ip rule add fwmark 102 table 102
    ip route add default via 10.0.3.2 table 102

    You need this to make packets go back through the right proxy. Otherwise you'll send those packets out via your router public interface which won't work.
     

  • WINDOWS: Setup hauthd to use multiple proxies and add servers appropriately to dbo.server table
  • WINDOWS: Make sure hauthd restarts every time L2 server restarts

 

Useful resources

 

 

Edited by eressea
  • Like 1

Share this post


Link to post
Share on other sites
bit    0

Would this work with single proxy and no router machine? I am kind of banging my head here trying to make that setup work.

 

My setup: 1 linux proxy running ppp server.

Proxy ppp0 interface with ip 192.168.0.1, windows server client with ip 192.168.0.234.

$ip in the commands is external proxy server ip.

 

iptables -t nat    -A PREROUTING         -p tcp -m multiport --dport 2106,7777        -j DNAT     --to-dest 192.168.0.234
iptables -t mangle -A PREROUTING -i eth0 -p tcp -m multiport --dport 2106,7777 -m tcp -j CONNMARK --set-mark 101
iptables -t mangle -A PREROUTING -i ppp0 -p tcp -m multiport --sport 2106,7777 -m tcp -j CONNMARK --restore-mark
ip rule add fwmark 101 table 101
ip route add default via $ip table 101

Any idea what i do wrong here?

 

This works, but server sees proxy IP as player IP:

iptables -t nat -A PREROUTING  -p tcp -m multiport --dport 2106,7777 -j DNAT       --to-dest 192.168.0.234
iptables -t nat -A POSTROUTING                                       -j MASQUERADE

 

By the way first post has some typos. Chain is missing in commands with "--restore-mark"

Share this post


Link to post
Share on other sites
eressea    12
52 minutes ago, bit said:

Would this work with single proxy and no router machine? I am kind of banging my head here trying to make that setup work.

 

My setup: 1 linux proxy running ppp server.

Proxy ppp0 interface with ip 192.168.0.1, windows server client with ip 192.168.0.234.

$ip in the commands is external proxy server ip.

 


iptables -t nat    -A PREROUTING         -p tcp -m multiport --dport 2106,7777        -j DNAT     --to-dest 192.168.0.234
iptables -t mangle -A PREROUTING -i eth0 -p tcp -m multiport --dport 2106,7777 -m tcp -j CONNMARK --set-mark 101
iptables -t mangle -A PREROUTING -i ppp0 -p tcp -m multiport --sport 2106,7777 -m tcp -j CONNMARK --restore-mark
ip rule add fwmark 101 table 101
ip route add default via $ip table 101

Any idea what i do wrong here?

 

This works, but server sees proxy IP as player IP:


iptables -t nat -A PREROUTING  -p tcp -m multiport --dport 2106,7777 -j DNAT       --to-dest 192.168.0.234
iptables -t nat -A POSTROUTING                                       -j MASQUERADE

 

By the way first post has some typos. Chain is missing in commands with "--restore-mark"

 

You can't do it without Linux router - Windows can't do policy-based routing.

 

You would have to set default gateway on Windows Server to 192.168.0.1 and make all packets out go through it (but you would lose connectivity to it from internet, so you would have to create static route via previous default gateway to your proxy; even that would be very inconvenient because you would have to access your server only from proxy or via the proxy - because there would be no other way to it).

 

Without right default gateway your only option is to use MASQUERADE or SNAT (it's the same, masquerade just automatically assumes new source IP address) which changes sender IP address to proxy IP - that you then see on Windows server.

Edited by eressea

Share this post


Link to post
Share on other sites
tiagopyerri    0
4 hours ago, eressea said:

HOWTO - L2 server with (transparent) proxies

DISCLAIMER: I wrote this HOWTO without really testing it (I've configured it for my server more than year ago), so if you find any mistake, feel free to comment/ask, I'll try to update this HOWTO appropriately.

Purpose of using proxies

The standard old way of setting up L2off server involves one Windows Server that is connected directly to internet and players connect directly to it's IP address. While it's enough for a small server, it may bring few problems:

  • If someone wants to DDoS your server, he just pays for some attack to your IP address. You're screwed.
  • There may be players who experience lags because route from their network to your server isn't perfect (because of bad peering between player's ISP and your datacenter etc). You can't just tell those players "Try better ISP".
  • Players from foreign countries may have good ping to datacenters in their country but not to your data center.

Some of these problems can be mitigated by using proxies for your server:

  • When someone buys DDoS attack to your login server on first proxy, you can just change IP address in your DNS and let your players login on your second proxy (you need to have domain name instead of IP address in your l2.ini).
  • When someone buys DDoS attack to your game server on first proxy, players can still play on second proxy.
  • When player has bad ping on first proxy, login will be bit slower but player can select second proxy with better ping and play on it

You can also do the same with your website, just replace ports 2106 and 7777 with 80 and 443...

What will you need

  • Some basic knowledge about L2off, Linux and networking. Also you'll have to read few manuals.
  • Linux router between your L2 server and internet, you can do it two ways (of course it must be configured to forward IPv4 traffic):
    • real physical Linux router between them
    • Linux as main operating system on your server hardware + L2 server in virtual machine (kvm-qemu).
  • Linux servers (1 core & 512 MB RAM is usually enough) for proxies
  • Copy of patched hauthd - READ INSTRUCTIONS THERE
  • Script that will be able to restart hauthd when L2 server crashes/exits

How to do it

I will use following IP networks and addresses in this example:

  • Networks:
    • Local network 10.0.0.0/24 between Linux router and Windows VM
    • Virtual network 10.0.1.0/24 between Linux router and proxies
  • Addresses:
    • 1.2.3.4 on Linux router eth0
    • 2.3.4.5 on first proxy eth0
    • 3.4.5.6 on second proxy eth0
    • 4.5.6.7 on third proxy eth0
    • 10.0.0.1 on Linux router eth1 (for physical router) or tap0 (for kvm-qemu variant)
    • 10.0.0.2 on Windows server Local network interface
    • 10.0.1.1 on Linux router tun0 (OpenVPN server 1)
    • 10.0.2.1 on Linux router tun1 (OpenVPN server 2)
    • 10.0.3.1 on Linux router tun2 (OpenVPN server 3)
    • 10.0.1.2 on first proxy tun0 (OpenVPN client)
    • 10.0.2.2 on second proxy tun0 (OpenVPN client)
    • 10.0.3.2 on third proxy tun0 (OpenVPN client)

 

Physical network:

 

l2-proxies-physical-network.png.aced8219e6456f630529db2754136d09.png

 

Logical network:

 

l2-proxies-logical-network.png.8dc15fb59fab157a254514dddca94364.png

 

Now proceed with installation and configuration:

  • ROUTER + PROXIES: Install OpenVPN
  • ROUTER: For each proxy, configure one instance of OpenVPN server with static key (it will have to run on different ports, for example 1194, 1195, 1196)
  • PROXY: Copy static key from router, configure OpenVPN client
  • WINDOWS: Make sure windows server can ping linux router (ping 10.0.0.1)
  • PROXY: Make sure proxies can ping server (ping 10.0.1.1 / ping 10.0.2.1 / ping 10.0.3.1)
  • PROXY: Setup NAT (find some way how to run it on every start!)
    
    iptables -t nat -A PREROUTING -m tcp -p tcp --dport 2106 -j DNAT --to-destination 10.0.0.2:2106
    iptables -t nat -A PREROUTING -m tcp -p tcp --dport 7777 -j DNAT --to-destination 10.0.0.2:7777
    
    You need this to make proxies forward traffic on ports 2106 and 7777 to windows server.
     
  • ROUTER: Setup connmark (find some way how to run it on every start! If it's physical router, use eth1 instead of tap0!):
    
    iptables -t mangle -A PREROUTING -i tun0 -p tcp -m tcp --dport 7777 -j CONNMARK --set-mark 100
    iptables -t mangle -A PREROUTING -i tun1 -p tcp -m tcp --dport 2106 -j CONNMARK --set-mark 101
    iptables -t mangle -A PREROUTING -i tun1 -p tcp -m tcp --dport 7777 -j CONNMARK --set-mark 101
    iptables -t mangle -A PREROUTING -i tun2 -p tcp -m tcp --dport 2106 -j CONNMARK --set-mark 101
    iptables -t mangle -A PREROUTING -i tun2 -p tcp -m tcp --dport 7777 -j CONNMARK --set-mark 101
    iptables -t mangle -i tap0 -p tcp -m tcp --sport 2106 -j CONNMARK --restore-mark
    iptables -t mangle -i tap0 -p tcp -m tcp --sport 7777 -j CONNMARK --restore-mark
    You need this to mark connections so you know where they came from.
     
  • ROUTER: Setup policy-based routing (find some way how to run it on every start!):
    
    ip rule add fwmark 100 table 100
    ip route add default via 10.0.1.2 table 100
    ip rule add fwmark 101 table 101
    ip route add default via 10.0.2.2 table 101
    ip rule add fwmark 102 table 102
    ip route add default via 10.0.3.2 table 102

    You need this to make packets go back through the right proxy. Otherwise you'll send those packets out via your router public interface which won't work.
     

  • WINDOWS: Setup hauthd to use multiple proxies and add servers appropriately to dbo.server table
  • WINDOWS: Make sure hauthd restarts every time L2 server restarts

 

Useful resources

 

 

and it's nice to have people who can teach others how to have a professional server these days, where servers do not exceed 90 days
 Deserves to be Fixed this topic

Share this post


Link to post
Share on other sites
bit    0

Actually i figured it out. Take a look.

 

Linux proxy server runs OpenVPN server.

OpenVPN network IP of proxy: 10.200.0.1

Windows connects to OpenVPN server, gets IP 10.200.0.10

AuthD/Server both listen on address 10.200.0.10

Do not set default gateway on windows server to 10.200.0.1 This is irrelevant. OpenVPN client does it automatically.

$ip as always - external IP of proxy server

Proxy firewall setup:

iptables        -A INPUT                 -s 10.200.0.0/24 -d 10.200.0.1                         -j ACCEPT
iptables -t nat -A POSTROUTING -o eth0   -s 10.200.0.0/24 -d 10.200.0.10                        -j SNAT --to $ip
iptables -t nat -A PREROUTING  -i eth0 ! -s 10.200.0.10   -p tcp -m multiport --dport 2106,7777 -j DNAT --to 10.200.0.10

To my surprise this setup server can see real client IP address. Also if default gateway is not set on windows everything still works and windows server can still access internet.

 

Can not try it with ppp yet because live server, but i suspect there would be no difference (except for lan IP addresses ofc).

 

Edit:

Turns out windows server indeed can not access internet. Still trying to figure out how to fix openvpn routes so internet works as well as proxied server.

 

Edit:

K i finally figured out how to make internet connection and forwarded server work as well. This is rather dirty solution but it works.

 

Key requirement is that you set default gateway to OpenVPN adapter on windows server. For some reason it is not done automatically.

Add following settings to your OpenVPN.ovpn client config file:

route-nopull
script-security 2
up-delay
up "setgw.bat"

Remove "redirect-gateway" option if you have it.

 

Create setgw.bat script in openvpn config folder with content:

start cmd /c "timeout 10 && netsh interface ipv4 set address %1 static %4 255.255.255.0 %5"

This file will sleep a bit and set default gateway to openvpn network adapter.

 

If you know better solution please share, because this one sucks.

Edited by bit

Share this post


Link to post
Share on other sites
eressea    12
17 hours ago, bit said:

If you know better solution please share, because this one sucks.

 

How about

  • install VirtualBox (or use Hyper-V but I don't have any experience with it)
  • create virtual machine with 1 core, 512 MB RAM and 10 GB disk image
  • install Debian Stretch with just only ssh server
  • configure networking so
    • host (windows) will have no IP address set on the primary NIC
    • guest (linux) will have eth0 bridged with windows primary NIC and will have IP address that your windows server had
    • guest (linux) will have eth1 just between host and guest and set some internal address on it (10.0.0.1/24)
    • host (windows) virtual NIC (linux eth1) will have internal address in the same segment (10.0.0.2/24) and default gateway set to linux IP (10.0.0.1)

Then you would have real linux router so you can do all the stuff needed (policy-based routing). But you need to find some way how to start that VM when windows start.

  • Upvote 1

Share this post


Link to post
Share on other sites
tk422    0

This is a great post. Thanks for taking the time to write it!

 

I would like to offer a few improvements as outlined below. This system i propose borrows heavily from my career life and experience. I've deployed and operate several of these systems and use one of my own to host professional business applications.

 

j9rnlY1.png

 

I intentionally did not provide step by step instructions on how to set this up. It is important that you actually spend the time to understand how the system works rather than blindly follow instructions. Nothing presented here is overly complex but it will likely require you to learn something new.

 

/// Resources

Online Diagram: L2 Server Hosting Environment

 

OVH Virtual Mac (Other providers offer this too!)
https://docs.ovh.com/gb/en/dedicated/network-virtual-mac/

 

MikroTik CHR
https://wiki.mikrotik.com/wiki/Manual:CHR

 

 

 

Edited by tk422

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.