Setting up a load balancer on slicehost

Goal: set-up a single load balancer using LVS-TUN technique to direct traffic to web servers running Rails being served by Passenger Phusion on Ubuntu 10.04.

1. Request slicehost to set-up a shared IP. The load balancer (LB) must have a shared IP with the web servers. Read the LVS-TUN docs to understand more of the why, but for the how, be like Arnold and just do it. If you explain to them what you're doing, they will know what to do and configure stuff the right way for you.

2. Install ldconfig

apt-get install ldirectord

3. Follow these directions ignoring the bits about heartbeat (I feel that heartbeat is needless complexity for me now but I'll probably want to add it later).

The Linux-Directors must be able to route traffic to the real-servers. Specifically in addition to correctly configuring the interfaces and routes you must enable IPV4 forwarding. This is done by adding configuration of net.ipv4.ip_forward to /etc/sysctl.conf as follows:

#



# Enables packet forwarding
net.ipv4.ip_forward = 1

For these changes to take effect the sysctl command may be used:

/sbin/sysctl -p
net.ipv4.ip_forward = 1

Thanks to UltraMonkey for this information.

4. Install ldirectord.cf in /etc. You can use the sample file from the UltraMonkey site article above. Make your modificaitons to reflect your setup. Mine looks like this:

checktimeout=10
checkinterval=2
autoreload=yes
logfile="local0"
quiescent=yes
 
virtual=SHARED.IP.ADDRESS:80
        real=web1.server.ip.address:80 ipip
        real=web2.22.33.44:80 ipip
        service=http
        request="healthcheck.html"
        receive="OK"
        scheduler=rr
        protocol=tcp
        checktype=negotiate


virtual=SHARED.IP.ADDRESS:443
        real=web1.server.ip.address:443 ipip
        real=web2.server.ip.address:443 ipip
        service=https
        request="healthcheck.html"
        receive="OK"
        scheduler=rr
        protocol=tcp
        checktype=negotiate

Obviously plug-in the correct IP addresses for the shared IP that you requested from slicehost, and the real IPs of the 2 web servers.

Check the output of ipvsadm:

root@SatelliteIngestionLB-01:/etc# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  67.207.134.92:80 rr
  -> 67.207.134.84:80             Tunnel  0      0          0         
  -> 67.207.143.51:80             Tunnel  0      0          0         
TCP  67.207.134.92:443 rr
  -> 67.207.134.84:443            Tunnel  0      0          0         
  -> 67.207.143.51:443            Tunnel  0      0          0  

5. ARP hacking

Follow the "Restrict ARP advertising" config on Ultra Monkey. I'll copy it here just in case but this is their genius not mine!

On Debian, the arp_ignore and arp_announce sysctl flags are used to ensure that the VIP is not announced via ARP on the real-servers, eth0 should be set to only use its addresses for ARP. This is done by adding the following lines to /etc/sysctl.conf. The eth0 lines below should be replicated for any devices that respond to ARP.

# Enable configuration of arp_ignore option
net.ipv4.conf.all.arp_ignore = 1

# When an arp request is received on eth0, only respond if that address is
# configured on eth0. In particular, do not respond if the address is
# configured on lo
net.ipv4.conf.eth0.arp_ignore = 1

# Ditto for eth1, add for all ARPing interfaces
#net.ipv4.conf.eth1.arp_ignore = 1


# Enable configuration of arp_announce option
net.ipv4.conf.all.arp_announce = 2

# When making an ARP request sent through eth0 Always use an address that
# is configured on eth0 as the source address of the ARP request.  If this
# is not set, and packets are being sent out eth0 for an address that is on
# lo, and an arp request is required, then the address on lo will be used.
# As the source IP address of arp requests is entered into the ARP cache on
# the destination, it has the effect of announcing this address.  This is
# not desirable in this case as adresses on lo on the real-servers should
# be announced only by the linux-director.
net.ipv4.conf.eth0.arp_announce = 2

# Ditto for eth1, add for all ARPing interfaces
#net.ipv4.conf.eth1.arp_announce = 2

Don't forget to run sysctl -p after editing this file.


6. Create a tunnel network interface

make sure ipip is enabled

modprobe ipip

then add the tunl0 interface to /etc/network/interfaces

iface tunl0 inet static
    pre-up ip tunnel add tunl0 mode ipip
    post-down ip tunnel del tunl0 mode ipip
    address 67.207.134.92
    netmask 255.255.255.255

then bring it up

ifup tunl0

see if it worked

/etc/network# ip add sh tunl0
7: tunl0:  mtu 1480 qdisc noqueue state UNKNOWN 
    link/ipip 0.0.0.0 brd 0.0.0.0
    inet 11.22.33.44/32 scope global tunl0

add a permanent route

route add -host 67.207.134.92 dev tunl0

ok almost works... still seeing only 443 on ipvsadm

 ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  67.207.134.92:443 rr
  -> 67.207.134.84:443            Tunnel  0      0          0         
  -> 67.207.143.51:443            Tunnel  0      0          0         

Fresh Stories

Bad stories are like dandelions. Once they get into the backlog, they tend to multiply. You can keep weeding them, but they'll usually come back. The thing is, there is no cure for them. In SDD, you have to just keep plucking out the crappy stories, keep dis-aggregating excessively vague and/or complex stories, checking with the team the whole time. Right now in the heat of development of a large complex project, the scrum master (myself) and the product owner are spending all day every day cleaning out bad stories, refreshing stale stories, writing new stories that offer greater clarity and detail (and therefore are more testable), consulting with the team about pointing the stories. It's a full time job for the two of us. Extending the definition of done requires that you have a high quality definition, and if you don't have a testable doneness criteria, you don't have a good story.

On the other hand, it's critical to remember that stories are (I think Mike Cohn said this) "a reminder to have a conversation". I've found that people new to agile constantly struggle with its intrinsic ambiguities and its reliance on a bit of chaos and emergence to sort itself out. In our project, we're handling this by keeping stories in current and at the top of the backlog in a very testable, verifiable that it is doneness state. They are typically 1 or 2 points on the 8 point scale, sometimes 3. A lot of the big system stories that are just big because of the heavy lifting they require behind the scenes are done (eg set-up the amazon s3 encrypted url generator or other things with low BV but high effort). As you go down the backlog, you see more and more dandelions - stale stories, too big stories, non-testable stories, etc. These are the reminder stories. In fact it seems like dandelions are a reminder to the scrum master and the product owner to do more work!

Something I've struggled with in the lack of hard and fast rules department is, when does a story have to deliver business value (BV) and when does it not? Is it ever ok to have "system" stories (hahaha, The system shall do everything I want in 3 weeks for $10)? I want philosophically for every story to deliver end-user BV, but in order to get there, we were ending up with 8 point stories that were just not helping the developers get stuff done. We argued about it, and we decided that there are some cases where the benefits of a non-BV story out weigh the potential drawbacks of "rewarding" the team with burn down for a story that is really a dependency for other stories to deliver BV. Setting up amazon cloudfront and s3 to use non-shareable urls for digital assets was a piece of architecture work that took a developer 3 days to do that then unblocked a large number of BV-delivering stories. We elected to point and burn down these stories, and I think the morale boost this generated was worth it. Sometimes the psychological needs of the team outweigh the benefit of philosophical purity, which can be intellectually interesting but not always pragmatic.

In conclusion, making agile work is all about being pragmatic most of the time, and reaching for the higher principals when it's a win. And it's a lot of freakin work. Keeping your backlog of stories accurately prioritized and testable is a full time job. If you're finding that your adoption of agile isn't working, ask yourself how many hours a day is your scrum master spending managing the backlog with the product owner. In the middle of a project, this could easily be 3 full business days of work a week.

7 tips on getting yet another new team up and running on Agile

Going to try to lay down some quick observations about getting new teams ramped up on Agile. It's just like everything else in life, you keep getting better at doing what you've been doing, but you keep having to lead new horses to the same water.

1. Agile is hard to understand, and it's really hard to understand if you aren't intimately familiar with the challenges of modern software development. When you have to bring in business stakeholders and other non-developer team members into the mix, be prepared for a longer than desirable ramp-up time. Reading books about Agile is like reading books about skiing. You can glimpse about 5% of what there is to know this way, but until you do it a lot of times for a lot of different projects, you ain't gone skiing yet.

2. Every project is different, and your process needs to be allowed to emerge. This is something that can drive new people crazy - "But how did you do it before on other projects? Why are we using process and procedure X when we did Y before?" People naturally resist emergent approaches because to them change means that there is a problem, that something is wrong, that something wasn't well thought out. They don't know that every project and every team is going to have an unique mix of personalities, technical problems, logistical problems, etc., that will result in an unique set of solutions that are built on the shoulders of giants from previous problems.

3. Initial story quality always sucks. Don't spent too much time doing it. You have to spend time and energy every iteration - for us, every day - evaluating existing stories, throwing stale ones out, and writing new stories that cover the gaps that you didn't think of until you tried to actually do any work.

4. Try to get from mockup to building real applications/web pages ASAP. No one will really understand what the true UX challenges are going to be with your brilliant ideas until you can click on something that does something. Light-weight mockup tools like Balsamiq are incredible productivity-enhancers for this phase, but don't spend too much time on them. YOU MUST BUILD to really discover. This is probably the most important and profound learning on my agile journey - Just Do It. Stop talking, stop thinking, stop planning, just build something, inspect what's wrong with it, solve one small well-scoped problem at a time. Emerge.

5. Do not confuse visual pretty makers with interaction designers. Pixel pushers almost universilly are bad at UX. If your product doesn't make sense to users it doesn't matter what it looks like. Decorate it later. The extra work your HTML person might have to do and bitch about is sad, but it's nothing compared to the amount time you saved not decorating first and building later.

6. Small bite size pieces at all levels, from stories, to the size of your iteration, to the time to initial launch, to your release cycle. The smaller you make things the less you have to think about, the more you can focus on doing one thing perfectly right.

7. There are two kinds of reactions to Agile. "I get it, this is how it's done," and "WTF???" Don't bother trying to lecture or explain to the latter crowd. Lead them to water and spend as little time as you can trying to convince them of anything. Some people are never going to play the game as its meant to be played 100%, but as long as they don't get in the way it's ok.

Maven JAXB 2.0 schemagen goal

I've been building a RESTful API client using Sun's jersey JAX-RS reference implementation. My approach has been to create a domain model out of Java POJOs. It felt like wasted effort to create an XSD that was going to be derived from a fluid RESTful API that I have no control over. It's easier to use the JAXB annotations to control marshalling and unmarshalling behavior from API calls. The XSD seemed like an artifact that added no value... Except that I now have a use case where customers would like to provide me with XML data to transmit up to said API. Life is easier if customers can provide me with the data in a format I can just get for free. In order to create a data specification, I wanted to generate an XSD from my domain model using schemagen. Figuring this is something I'll want to do on a regular basis, and it's something other developers on the team may need to do in the future, I decided to use the Maven JAXB schemagen plugin, which is very poorly documented. However, I found a sample POM configuration that got me through quickly. Thanks to Mark for sharing! Just in case, I'll repost here.

<pluginrepositories>
   <pluginrepository>
           <id>maven-repository.dev.java.net</id>
           <name>Java.net Maven 2 Repository</name>
           <url>http://download.java.net/maven/2</url>
   </pluginrepository>
 </pluginrepositories>

      <plugin>
           <groupid>com.sun.tools.jxc.maven2</groupid>
           <artifactid>maven-jaxb-schemagen-plugin</artifactid>
           <version>1.2</version>
           <executions>
               <execution>
                   <phase>generate-resources</phase>
                    <goals>
                       <goal>generate</goal>
                    </goals>
              </execution>
          </executions>

           <configuration>
               <project>${project}</project>
               <destdir>${project.build.directory}/schemas</destdir>
               <srcdir>${project.build.sourceDirectory}/za/co/jumpingbean/dal/model/core/</srcdir>
               <verbose>true</verbose>
           </configuration>
          <dependencies>
               <dependency>
                   <groupid>javax.xml.bind</groupid>
                   <artifactid>jaxb-api</artifactid>
                   <version>2.2</version>
               </dependency>
               <dependency>
                   <groupid>com.sun.xml.bind</groupid>
                   <artifactid>jaxb-xjc</artifactid>
                   <version>2.2</version>
               </dependency>
               <dependency>
                   <groupid>com.sun.xml.bind</groupid>
                   <artifactid>jaxb-impl</artifactid>
                   <version>2.2</version>
               </dependency>
               <dependency>
                   <groupid>com.sun.xml.bind</groupid>
                   <artifactid>jaxb-xjc</artifactid>
                   <version>2.2</version>
               </dependency>
          </dependencies>
     </plugin>