Sunday, September 26, 2010

Setting up a load balancer on slicehost

Goal: set-up a single load balancer using LVS-TUN technique to direct traffic to web servers running Rails being served by Passenger Phusion on Ubuntu 10.04.

1. Request slicehost to set-up a shared IP. The load balancer (LB) must have a shared IP with the web servers. Read the LVS-TUN docs to understand more of the why, but for the how, be like Arnold and just do it. If you explain to them what you're doing, they will know what to do and configure stuff the right way for you.

2. Install ldconfig
apt-get install ldirectord


3. Follow these directions ignoring the bits about heartbeat (I feel that heartbeat is needless complexity for me now but I'll probably want to add it later).

The Linux-Directors must be able to route traffic to the real-servers. Specifically in addition to correctly configuring the interfaces and routes you must enable IPV4 forwarding. This is done by adding configuration of net.ipv4.ip_forward to /etc/sysctl.conf as follows:


#



# Enables packet forwarding
net.ipv4.ip_forward = 1

For these changes to take effect the sysctl command may be used:

/sbin/sysctl -p
net.ipv4.ip_forward = 1



Thanks to UltraMonkey for this information.

4. Install ldirectord.cf in /etc. You can use the sample file from the UltraMonkey site article above. Make your modificaitons to reflect your setup. Mine looks like this:


checktimeout=10
checkinterval=2
autoreload=yes
logfile="local0"
quiescent=yes

virtual=SHARED.IP.ADDRESS:80
real=web1.server.ip.address:80 ipip
real=web2.22.33.44:80 ipip
service=http
request="healthcheck.html"
receive="OK"
scheduler=rr
protocol=tcp
checktype=negotiate


virtual=SHARED.IP.ADDRESS:443
real=web1.server.ip.address:443 ipip
real=web2.server.ip.address:443 ipip
service=https
request="healthcheck.html"
receive="OK"
scheduler=rr
protocol=tcp
checktype=negotiate


Obviously plug-in the correct IP addresses for the shared IP that you requested from slicehost, and the real IPs of the 2 web servers.

Check the output of ipvsadm:


root@SatelliteIngestionLB-01:/etc# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 67.207.134.92:80 rr
-> 67.207.134.84:80 Tunnel 0 0 0
-> 67.207.143.51:80 Tunnel 0 0 0
TCP 67.207.134.92:443 rr
-> 67.207.134.84:443 Tunnel 0 0 0
-> 67.207.143.51:443 Tunnel 0 0 0


5. ARP hacking

Follow the "Restrict ARP advertising" config on Ultra Monkey. I'll copy it here just in case but this is their genius not mine!

On Debian, the arp_ignore and arp_announce sysctl flags are used to ensure that the VIP is not announced via ARP on the real-servers, eth0 should be set to only use its addresses for ARP. This is done by adding the following lines to /etc/sysctl.conf. The eth0 lines below should be replicated for any devices that respond to ARP.


# Enable configuration of arp_ignore option
net.ipv4.conf.all.arp_ignore = 1

# When an arp request is received on eth0, only respond if that address is
# configured on eth0. In particular, do not respond if the address is
# configured on lo
net.ipv4.conf.eth0.arp_ignore = 1

# Ditto for eth1, add for all ARPing interfaces
#net.ipv4.conf.eth1.arp_ignore = 1


# Enable configuration of arp_announce option
net.ipv4.conf.all.arp_announce = 2

# When making an ARP request sent through eth0 Always use an address that
# is configured on eth0 as the source address of the ARP request. If this
# is not set, and packets are being sent out eth0 for an address that is on
# lo, and an arp request is required, then the address on lo will be used.
# As the source IP address of arp requests is entered into the ARP cache on
# the destination, it has the effect of announcing this address. This is
# not desirable in this case as adresses on lo on the real-servers should
# be announced only by the linux-director.
net.ipv4.conf.eth0.arp_announce = 2

# Ditto for eth1, add for all ARPing interfaces
#net.ipv4.conf.eth1.arp_announce = 2


Don't forget to run sysctl -p after editing this file.


6. Create a tunnel network interface

make sure ipip is enabled


modprobe ipip


then add the tunl0 interface to /etc/network/interfaces


iface tunl0 inet static
pre-up ip tunnel add tunl0 mode ipip
post-down ip tunnel del tunl0 mode ipip
address 67.207.134.92
netmask 255.255.255.255


then bring it up


ifup tunl0


see if it worked


/etc/network# ip add sh tunl0
7: tunl0: mtu 1480 qdisc noqueue state UNKNOWN
link/ipip 0.0.0.0 brd 0.0.0.0
inet 11.22.33.44/32 scope global tunl0


add a permanent route


route add -host 67.207.134.92 dev tunl0


ok almost works... still seeing only 443 on ipvsadm


ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 67.207.134.92:443 rr
-> 67.207.134.84:443 Tunnel 0 0 0
-> 67.207.143.51:443 Tunnel 0 0 0