Setting up a load balancer on slicehost

Goal: set-up a single load balancer using LVS-TUN technique to direct traffic to web servers running Rails being served by Passenger Phusion on Ubuntu 10.04.

1. Request slicehost to set-up a shared IP. The load balancer (LB) must have a shared IP with the web servers. Read the LVS-TUN docs to understand more of the why, but for the how, be like Arnold and just do it. If you explain to them what you're doing, they will know what to do and configure stuff the right way for you.

2. Install ldconfig

apt-get install ldirectord

3. Follow these directions ignoring the bits about heartbeat (I feel that heartbeat is needless complexity for me now but I'll probably want to add it later).

The Linux-Directors must be able to route traffic to the real-servers. Specifically in addition to correctly configuring the interfaces and routes you must enable IPV4 forwarding. This is done by adding configuration of net.ipv4.ip_forward to /etc/sysctl.conf as follows:

#



# Enables packet forwarding
net.ipv4.ip_forward = 1

For these changes to take effect the sysctl command may be used:

/sbin/sysctl -p
net.ipv4.ip_forward = 1

Thanks to UltraMonkey for this information.

4. Install ldirectord.cf in /etc. You can use the sample file from the UltraMonkey site article above. Make your modificaitons to reflect your setup. Mine looks like this:

checktimeout=10
checkinterval=2
autoreload=yes
logfile="local0"
quiescent=yes
 
virtual=SHARED.IP.ADDRESS:80
        real=web1.server.ip.address:80 ipip
        real=web2.22.33.44:80 ipip
        service=http
        request="healthcheck.html"
        receive="OK"
        scheduler=rr
        protocol=tcp
        checktype=negotiate


virtual=SHARED.IP.ADDRESS:443
        real=web1.server.ip.address:443 ipip
        real=web2.server.ip.address:443 ipip
        service=https
        request="healthcheck.html"
        receive="OK"
        scheduler=rr
        protocol=tcp
        checktype=negotiate

Obviously plug-in the correct IP addresses for the shared IP that you requested from slicehost, and the real IPs of the 2 web servers.

Check the output of ipvsadm:

root@SatelliteIngestionLB-01:/etc# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  67.207.134.92:80 rr
  -> 67.207.134.84:80             Tunnel  0      0          0         
  -> 67.207.143.51:80             Tunnel  0      0          0         
TCP  67.207.134.92:443 rr
  -> 67.207.134.84:443            Tunnel  0      0          0         
  -> 67.207.143.51:443            Tunnel  0      0          0  

5. ARP hacking

Follow the "Restrict ARP advertising" config on Ultra Monkey. I'll copy it here just in case but this is their genius not mine!

On Debian, the arp_ignore and arp_announce sysctl flags are used to ensure that the VIP is not announced via ARP on the real-servers, eth0 should be set to only use its addresses for ARP. This is done by adding the following lines to /etc/sysctl.conf. The eth0 lines below should be replicated for any devices that respond to ARP.

# Enable configuration of arp_ignore option
net.ipv4.conf.all.arp_ignore = 1

# When an arp request is received on eth0, only respond if that address is
# configured on eth0. In particular, do not respond if the address is
# configured on lo
net.ipv4.conf.eth0.arp_ignore = 1

# Ditto for eth1, add for all ARPing interfaces
#net.ipv4.conf.eth1.arp_ignore = 1


# Enable configuration of arp_announce option
net.ipv4.conf.all.arp_announce = 2

# When making an ARP request sent through eth0 Always use an address that
# is configured on eth0 as the source address of the ARP request.  If this
# is not set, and packets are being sent out eth0 for an address that is on
# lo, and an arp request is required, then the address on lo will be used.
# As the source IP address of arp requests is entered into the ARP cache on
# the destination, it has the effect of announcing this address.  This is
# not desirable in this case as adresses on lo on the real-servers should
# be announced only by the linux-director.
net.ipv4.conf.eth0.arp_announce = 2

# Ditto for eth1, add for all ARPing interfaces
#net.ipv4.conf.eth1.arp_announce = 2

Don't forget to run sysctl -p after editing this file.


6. Create a tunnel network interface

make sure ipip is enabled

modprobe ipip

then add the tunl0 interface to /etc/network/interfaces

iface tunl0 inet static
    pre-up ip tunnel add tunl0 mode ipip
    post-down ip tunnel del tunl0 mode ipip
    address 67.207.134.92
    netmask 255.255.255.255

then bring it up

ifup tunl0

see if it worked

/etc/network# ip add sh tunl0
7: tunl0:  mtu 1480 qdisc noqueue state UNKNOWN 
    link/ipip 0.0.0.0 brd 0.0.0.0
    inet 11.22.33.44/32 scope global tunl0

add a permanent route

route add -host 67.207.134.92 dev tunl0

ok almost works... still seeing only 443 on ipvsadm

 ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  67.207.134.92:443 rr
  -> 67.207.134.84:443            Tunnel  0      0          0         
  -> 67.207.143.51:443            Tunnel  0      0          0