tag:blogger.com,1999:blog-67242963572822566472024-03-08T07:07:31.409-05:00simple constraintsThis blog is my developer and manager's notebook to remember all the amazing great and interesting things I am learningUnknownnoreply@blogger.comBlogger26125tag:blogger.com,1999:blog-6724296357282256647.post-30398412668894251122010-09-26T00:09:00.003-04:002010-09-26T04:40:11.552-04:00Setting up a load balancer on slicehostGoal: set-up a single load balancer using <a href="http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.LVS-Tun.html#LVS-Tun-intro">LVS-TUN</a> technique to direct traffic to web servers running Rails being served by Passenger Phusion on Ubuntu 10.04.<br /><br />1. Request slicehost to set-up a shared IP. The load balancer (LB) must have a shared IP with the web servers. Read the LVS-TUN docs to understand more of the why, but for the how, be like Arnold and just do it. If you explain to them what you're doing, they will know what to do and configure stuff the right way for you.<br /><br />2. Install ldconfig <pre>apt-get install ldirectord</pre><br /><br />3. Follow these directions ignoring the bits about heartbeat (I feel that heartbeat is needless complexity for me now but I'll probably want to add it later). <br /><br />The Linux-Directors must be able to route traffic to the real-servers. Specifically in addition to correctly configuring the interfaces and routes you must enable IPV4 forwarding. This is done by adding configuration of net.ipv4.ip_forward to /etc/sysctl.conf as follows:<br /><br /><pre><br />#<br /><br /><br /><br /># Enables packet forwarding<br />net.ipv4.ip_forward = 1<br /><br />For these changes to take effect the sysctl command may be used:<br /><br />/sbin/sysctl -p<br />net.ipv4.ip_forward = 1<br /><br /></pre><br /><br />Thanks to <a href="http://www.ultramonkey.org/3/topologies/hc-ha-lb-eg.html">UltraMonkey</a> for this information. <br /><br />4. Install ldirectord.cf in /etc. You can use the sample file from the UltraMonkey site article above. Make your modificaitons to reflect your setup. Mine looks like this:<br /><br /><pre><br />checktimeout=10<br />checkinterval=2<br />autoreload=yes<br />logfile="local0"<br />quiescent=yes<br /> <br />virtual=SHARED.IP.ADDRESS:80<br /> real=web1.server.ip.address:80 ipip<br /> real=web2.22.33.44:80 ipip<br /> service=http<br /> request="healthcheck.html"<br /> receive="OK"<br /> scheduler=rr<br /> protocol=tcp<br /> checktype=negotiate<br /><br /><br />virtual=SHARED.IP.ADDRESS:443<br /> real=web1.server.ip.address:443 ipip<br /> real=web2.server.ip.address:443 ipip<br /> service=https<br /> request="healthcheck.html"<br /> receive="OK"<br /> scheduler=rr<br /> protocol=tcp<br /> checktype=negotiate<br /></pre><br /><br />Obviously plug-in the correct IP addresses for the shared IP that you requested from slicehost, and the real IPs of the 2 web servers. <br /><br />Check the output of ipvsadm:<br /><br /><pre><br />root@SatelliteIngestionLB-01:/etc# ipvsadm -L -n<br />IP Virtual Server version 1.2.1 (size=4096)<br />Prot LocalAddress:Port Scheduler Flags<br /> -> RemoteAddress:Port Forward Weight ActiveConn InActConn<br />TCP 67.207.134.92:80 rr<br /> -> 67.207.134.84:80 Tunnel 0 0 0 <br /> -> 67.207.143.51:80 Tunnel 0 0 0 <br />TCP 67.207.134.92:443 rr<br /> -> 67.207.134.84:443 Tunnel 0 0 0 <br /> -> 67.207.143.51:443 Tunnel 0 0 0 <br /></pre><br /><br />5. ARP hacking<br /><br />Follow the "Restrict ARP advertising" config on Ultra Monkey. I'll copy it here just in case but this is their genius not mine!<br /><br />On Debian, the arp_ignore and arp_announce sysctl flags are used to ensure that the VIP is not announced via ARP on the real-servers, eth0 should be set to only use its addresses for ARP. This is done by adding the following lines to /etc/sysctl.conf. The eth0 lines below should be replicated for any devices that respond to ARP. <br /><br /><pre><br /># Enable configuration of arp_ignore option<br />net.ipv4.conf.all.arp_ignore = 1<br /><br /># When an arp request is received on eth0, only respond if that address is<br /># configured on eth0. In particular, do not respond if the address is<br /># configured on lo<br />net.ipv4.conf.eth0.arp_ignore = 1<br /><br /># Ditto for eth1, add for all ARPing interfaces<br />#net.ipv4.conf.eth1.arp_ignore = 1<br /><br /><br /># Enable configuration of arp_announce option<br />net.ipv4.conf.all.arp_announce = 2<br /><br /># When making an ARP request sent through eth0 Always use an address that<br /># is configured on eth0 as the source address of the ARP request. If this<br /># is not set, and packets are being sent out eth0 for an address that is on<br /># lo, and an arp request is required, then the address on lo will be used.<br /># As the source IP address of arp requests is entered into the ARP cache on<br /># the destination, it has the effect of announcing this address. This is<br /># not desirable in this case as adresses on lo on the real-servers should<br /># be announced only by the linux-director.<br />net.ipv4.conf.eth0.arp_announce = 2<br /><br /># Ditto for eth1, add for all ARPing interfaces<br />#net.ipv4.conf.eth1.arp_announce = 2<br /></pre><br /><br />Don't forget to run sysctl -p after editing this file.<br /><br /><br />6. Create a tunnel network interface<br /><br />make sure ipip is enabled<br /><br /><pre><br />modprobe ipip<br /></pre><br /><br />then add the tunl0 interface to /etc/network/interfaces<br /><br /><pre><br />iface tunl0 inet static<br /> pre-up ip tunnel add tunl0 mode ipip<br /> post-down ip tunnel del tunl0 mode ipip<br /> address 67.207.134.92<br /> netmask 255.255.255.255<br /></pre><br /><br />then bring it up<br /><br /><pre><br />ifup tunl0<br /></pre><br /><br />see if it worked<br /><br /><pre><br />/etc/network# ip add sh tunl0<br />7: tunl0: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN <br /> link/ipip 0.0.0.0 brd 0.0.0.0<br /> inet 11.22.33.44/32 scope global tunl0<br /></pre><br /><br />add a permanent route<br /><br /><pre><br />route add -host 67.207.134.92 dev tunl0<br /></pre><br /><br />ok almost works... still seeing only 443 on ipvsadm<br /><br /><pre><br /> ipvsadm -L -n<br />IP Virtual Server version 1.2.1 (size=4096)<br />Prot LocalAddress:Port Scheduler Flags<br /> -> RemoteAddress:Port Forward Weight ActiveConn InActConn<br />TCP 67.207.134.92:443 rr<br /> -> 67.207.134.84:443 Tunnel 0 0 0 <br /> -> 67.207.143.51:443 Tunnel 0 0 0 <br /></pre>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6724296357282256647.post-63996334734508834862010-06-17T15:13:00.002-04:002010-06-17T15:24:22.109-04:00Fresh StoriesBad stories are like dandelions. Once they get into the backlog, they tend to multiply. You can keep weeding them, but they'll usually come back. The thing is, there is no cure for them. In SDD, you have to just keep plucking out the crappy stories, keep dis-aggregating excessively vague and/or complex stories, checking with the team the whole time. Right now in the heat of development of a large complex project, the scrum master (myself) and the product owner are spending all day every day cleaning out bad stories, refreshing stale stories, writing new stories that offer greater clarity and detail (and therefore are more testable), consulting with the team about pointing the stories. It's a full time job for the two of us. Extending the definition of done requires that you have a high quality definition, and if you don't have a testable doneness criteria, you don't have a good story.<div><br /></div><div>On the other hand, it's critical to remember that stories are (I think Mike Cohn said this) "a reminder to have a conversation". I've found that people new to agile constantly struggle with its intrinsic ambiguities and its reliance on a bit of chaos and emergence to sort itself out. In our project, we're handling this by keeping stories in current and at the top of the backlog in a very testable, verifiable that it is doneness state. They are typically 1 or 2 points on the 8 point scale, sometimes 3. A lot of the big system stories that are just big because of the heavy lifting they require behind the scenes are done (eg set-up the amazon s3 encrypted url generator or other things with low BV but high effort). As you go down the backlog, you see more and more dandelions - stale stories, too big stories, non-testable stories, etc. These are the reminder stories. In fact it seems like dandelions are a reminder to the scrum master and the product owner to do more work! </div><div><br /></div><div>Something I've struggled with in the lack of hard and fast rules department is, when does a story have to deliver business value (BV) and when does it not? Is it ever ok to have "system" stories (hahaha, The system shall do everything I want in 3 weeks for $10)? I want philosophically for every story to deliver end-user BV, but in order to get there, we were ending up with 8 point stories that were just not helping the developers get stuff done. We argued about it, and we decided that there are some cases where the benefits of a non-BV story out weigh the potential drawbacks of "rewarding" the team with burn down for a story that is really a dependency for other stories to deliver BV. Setting up amazon cloudfront and s3 to use non-shareable urls for digital assets was a piece of architecture work that took a developer 3 days to do that then unblocked a large number of BV-delivering stories. We elected to point and burn down these stories, and I think the morale boost this generated was worth it. Sometimes the psychological needs of the team outweigh the benefit of philosophical purity, which can be intellectually interesting but not always pragmatic.</div><div><br /></div><div>In conclusion, making agile work is all about being pragmatic most of the time, and reaching for the higher principals when it's a win. And it's a lot of freakin work. Keeping your backlog of stories accurately prioritized and testable is a full time job. If you're finding that your adoption of agile isn't working, ask yourself how many hours a day is your scrum master spending managing the backlog with the product owner. In the middle of a project, this could easily be 3 full business days of work a week. </div><div><br /></div><div><br /></div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6724296357282256647.post-75897611105284471822010-06-04T16:33:00.002-04:002010-06-04T16:47:49.431-04:007 tips on getting yet another new team up and running on AgileGoing to try to lay down some quick observations about getting new teams ramped up on Agile. It's just like everything else in life, you keep getting better at doing what you've been doing, but you keep having to lead new horses to the same water.<br /><br />1. Agile is hard to understand, and it's really hard to understand if you aren't intimately familiar with the challenges of modern software development. When you have to bring in business stakeholders and other non-developer team members into the mix, be prepared for a longer than desirable ramp-up time. Reading books about Agile is like reading books about skiing. You can glimpse about 5% of what there is to know this way, but until you do it a lot of times for a lot of different projects, you ain't gone skiing yet. <br /><br />2. Every project is different, and your process needs to be allowed to emerge. This is something that can drive new people crazy - "But how did you do it before on other projects? Why are we using process and procedure X when we did Y before?" People naturally resist emergent approaches because to them change means that there is a problem, that something is wrong, that something wasn't well thought out. They don't know that every project and every team is going to have an unique mix of personalities, technical problems, logistical problems, etc., that will result in an unique set of solutions that are built on the shoulders of giants from previous problems.<br /><br />3. Initial story quality always sucks. Don't spent too much time doing it. You have to spend time and energy every iteration - for us, every day - evaluating existing stories, throwing stale ones out, and writing new stories that cover the gaps that you didn't think of until you tried to actually do any work.<br /><br />4. Try to get from mockup to building real applications/web pages ASAP. No one will really understand what the true UX challenges are going to be with your brilliant ideas until you can click on something that does something. Light-weight mockup tools like <a href="http://www.balsamiq.com/">Balsamiq</a> are incredible productivity-enhancers for this phase, but don't spend too much time on them. YOU MUST BUILD to really discover. This is probably the most important and profound learning on my agile journey - Just Do It. Stop talking, stop thinking, stop planning, just build something, inspect what's wrong with it, solve one small well-scoped problem at a time. Emerge. <br /><br />5. Do not confuse visual pretty makers with interaction designers. Pixel pushers almost universilly are bad at UX. If your product doesn't make sense to users it doesn't matter what it looks like. Decorate it later. The extra work your HTML person might have to do and bitch about is sad, but it's nothing compared to the amount time you saved not decorating first and building later. <br /><br />6. Small bite size pieces at all levels, from stories, to the size of your iteration, to the time to initial launch, to your release cycle. The smaller you make things the less you have to think about, the more you can focus on doing one thing perfectly right.<br /><br />7. There are two kinds of reactions to Agile. "I get it, this is how it's done," and "WTF???" Don't bother trying to lecture or explain to the latter crowd. Lead them to water and spend as little time as you can trying to convince them of anything. Some people are never going to play the game as its meant to be played 100%, but as long as they don't get in the way it's ok.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6724296357282256647.post-37375117640324523752010-04-06T15:32:00.002-04:002010-04-06T15:41:53.217-04:00Maven JAXB 2.0 schemagen goalI've been building a RESTful API client using Sun's jersey JAX-RS reference implementation. My approach has been to create a domain model out of Java POJOs. It felt like wasted effort to create an XSD that was going to be derived from a fluid RESTful API that I have no control over. It's easier to use the JAXB annotations to control marshalling and unmarshalling behavior from API calls. The XSD seemed like an artifact that added no value... Except that I now have a use case where customers would like to provide me with XML data to transmit up to said API. Life is easier if customers can provide me with the data in a format I can just get for free. In order to create a data specification, I wanted to generate an XSD from my domain model using schemagen. Figuring this is something I'll want to do on a regular basis, and it's something other developers on the team may need to do in the future, I decided to use the Maven JAXB schemagen plugin, which is very poorly documented. However, I found <a href="http://www.jumpingbean.co.za/blog/maven-jaxb-schemagen-errors-pom.xml">a sample POM configuration</a> that got me through quickly. Thanks to Mark for sharing! Just in case, I'll repost here.<br /><br /><pre><br /><pluginrepositories><br /> <pluginrepository><br /> <id>maven-repository.dev.java.net</id><br /> <name>Java.net Maven 2 Repository</name><br /> <url>http://download.java.net/maven/2</url><br /> </pluginrepository><br /> </pluginrepositories><br /><br /> <plugin><br /> <groupid>com.sun.tools.jxc.maven2</groupid><br /> <artifactid>maven-jaxb-schemagen-plugin</artifactid><br /> <version>1.2</version><br /> <executions><br /> <execution><br /> <phase>generate-resources</phase><br /> <goals><br /> <goal>generate</goal><br /> </goals><br /> </execution><br /> </executions><br /><br /> <configuration><br /> <project>${project}</project><br /> <destdir>${project.build.directory}/schemas</destdir><br /> <srcdir>${project.build.sourceDirectory}/za/co/jumpingbean/dal/model/core/</srcdir><br /> <verbose>true</verbose><br /> </configuration><br /> <dependencies><br /> <dependency><br /> <groupid>javax.xml.bind</groupid><br /> <artifactid>jaxb-api</artifactid><br /> <version>2.2</version><br /> </dependency><br /> <dependency><br /> <groupid>com.sun.xml.bind</groupid><br /> <artifactid>jaxb-xjc</artifactid><br /> <version>2.2</version><br /> </dependency><br /> <dependency><br /> <groupid>com.sun.xml.bind</groupid><br /> <artifactid>jaxb-impl</artifactid><br /> <version>2.2</version><br /> </dependency><br /> <dependency><br /> <groupid>com.sun.xml.bind</groupid><br /> <artifactid>jaxb-xjc</artifactid><br /> <version>2.2</version><br /> </dependency><br /> </dependencies><br /> </plugin><br /></pre>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6724296357282256647.post-44190051300813702672009-04-17T13:54:00.003-04:002009-04-17T14:37:02.388-04:00What is strategy?Strategy is the process of choosing a broad goal, creating measurable objectives, executing a plan, and seeing if your execution is driving towards the objectives. It sounds incredibly easy, yet very few people are capable of actually conducting themselves in this manner in a professional context. You could ask many reasons why, but that wouldn't help you make the change. <br /><br />Agile development methodologies are all about coming up with short, concise, measurable objectives in the form of a lightweight spec, such as a user story. I don't know if it is beneficial for team members to really get their head around the mechanics of how an effective strategy works. I think one of the outputs of effective strategic execution is predictable results, such as being able to forecast a team's velocity.<br /><br />I'll tell a little parable that illustrates how even in the simplest of situations, our emotions and other psychological inputs can distract us from executing our strategy. Anyone who knows anyone from New York City and its environs will be amazed to find that New Yorkers talk about trains the way people in other cities talk about traffic. And anyone who lives in the boroughs is required to understand the nuance and intricacies of the train schedules on the weekend. <br /><br />My fiancee has been spending a lot of time with me in NYC and I've had to explain all of my strategies to her for effective train riding. Check out the <a href="http://www.mta.info/nyct/maps/submap.htm">system map</a> if you're not familiar. Our apartment is in Ft. Greene right near one of the busiest subway stations, Atlantic-Pacific. From our house you are within 3 blocks of the A, B, C, D, Q, G, M, N, R, 2, 3, 4, and 5 trains! So the question is often, which train should I take? Well, the entrance that is closest to our house takes you right down to the platform where the B/Q can be caught, so we almost always take these two lines. <br /><br />So that right there is an example of strategy in action: our goal is to wander the least and catch a train in the least amount of time that will get us to our destination in the least amount of time. The N and the Q are probably the 2 best trains that go through the station, but a stupid strategy would be to go downstairs and if the Q isn't there, go get the N - this is because in the time it takes you to walk to the N, you will probably miss the B or the Q. If I am trying to go somewhere that is only serviced by the Broadway line, I will still just wait for the Q (unless of course it is a weekend in which case I will walk over to the D/N platform since those trains, while requiring an extra walk, are more reliable on the weekend). <br /><br />Now my office is over on 26th between 6th and 7th. The very closest station is in fact the 1 train station at 7th ave and 28th street. But the 2/3 is really slow and I have to transfer from the 2/3 to the 1, so this route is automatically out of the running. The next closest is the F train station at 23rd and 6th, so if I catch the B train, I have a strategy: if I pull into Broadway/Lafayette or West 4th, and the F train is pulling in at the exact same time, such that I can catch the F with no waiting, it is faster to take the F. Otherwise I just stay on the B and take it up to 34th street, the next stop after W. 4th. While this is a longer walk, waiting for the F is longer.<br /><br />Does this make any sense? I've optimized my journey for the MOST important objective, which is to get to work in the least time possible so that I can sleep later and eat breakfast at home which is healthier than eating a bagel and cream cheese from the deli. The objective of getting to the station that is closest to my office is less important than getting to my office quickly. Doing the extra walk from 34th street also serves a nice-to-have objective of trying to get as much exercise as I can even when I don't go to the gym.<br /><br />There are myriad choices to get from my house to the office, and all of them will eventually result in my arrival at work; however, I've clearly identified an objective, which is different from a goal since it is measurable, which is to get to work in the least amount of time possible for the least amount of money. There are many decision points along the way, but by limiting the number of branches in the decision-making process, I have arrived at a very consistent formula. If I take the B or the Q, I know I can be at work, door-to-door, in about 40 minutes. Taking the F may reduce this time by a few minutes, but the risk of sitting on the F train platform for 5 minutes (which happens maybe 1 out of 3 times I choose to do so) is not a risk worth taking. <br /><br />Teams doing work need to limit their choices and come up with a consistent tactic to getting work done quickly by choosing the right goals and objectives. Over and over again, I find the most effective way to trim time off of projects getting done is to force developers to give me code that I can acceptance test on the build host. I generally don't consider any of the sub-stories of value, I only care about stories like, "As the public, I can see this website feature do something." Plumbing may have been required to get there, but the surest way to know if the plumbing, chores, sysadmin crap, and all other work required to get paid - because that is the mother of all objectives - is if I have a working website that lets me exercise the story in question. Letting broken untestable or unacceptable (meaning work that can not be acceptance tested) sit around is classic waste in the lean sense of the word.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6724296357282256647.post-9995418579435751692009-04-15T11:57:00.002-04:002009-04-15T12:06:37.966-04:00Agile ContractingOh man, this topic is a doozy. Customers want to know how much something will cost and when it will be done without knowing what it is we are really going to end up doing. When you suggest that we will discover things and course correct, they stare at you blankly and ask if that means they are getting a discount. Customers generally don't want to have to think, and they want to shift the responsibility of designing their business model onto you, the contractor, because bless their hearts, they probably don't even have a business plan that would let them articulate their business and revenue model as it applies to the work they want you to do. <br /><br />Having a consulting services based business means that your business consulting service model must be laced up and iron clad. You need to be the Delta Force that your customer is looking for. When you are a small business trying to keep all your customers happy, doing operations and support, figuring out your own revenue model that's required to support your existing client base, while at the same time figuring out how to grow your company to cover your new expenses that have emerged from your own success... it's really hard to do this. This is why most small businesses fail. An added challenge is when your business is small and there are limited numbers of partners, the partners tend to be the business development team, the sales team, the account execs, the PMs... and those roles have conflicting agendas that can make course correction in the middle of a project very hard. Sales people don't want to admit that the company's capabilities are a work in progress. <br /><br />Here are some very cool articles on alternative contract models. The fixed cost, fixed scope, fixed time model ALWAYS fails without a change control officer pounding the client relentlessly, which often results in an unhappy client; on the other hand, if you don't pound them with change control, they eat into your margin, for us, typically by 50% of our margin. We desperately want to be fixed scope bucket, but varying content of bucket; we want to be fixed cost and fixed time, where we append the emerging requirements into a new bucket of cost and time. <br /><br /><a href="http://www.sparkboxx.com/sparkboxx/types-of-contracts.html">http://www.sparkboxx.com/sparkboxx/types-of-contracts.html</a><br /><a href="http://alistair.cockburn.us/Agile+contracts">http://alistair.cockburn.us/Agile+contracts</a>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6724296357282256647.post-81823213288327345402009-04-10T17:39:00.003-04:002009-04-10T18:03:26.460-04:00Rails has no application contextSomething my developers have been complaining about lately is the difficulty in creating a generic dependency injection framework for a project with a lot of different plug-ins. I am asserting that there should be a generalized mechanism to set properties on a Rails domain model, rather than allowing plug-ins unholy access to reek havoc on my model objects.<br /><br />We have a use case where we want to be able to inject different implementations for the same interface. The idea is that we are using attachment_fu to decorate models, but we want to be able to store attachments in a different persistence mechanism (currently local filesystem or Amazon S3). We've implemented a generic framework as an engine that depends on attachment_fu to handle the attachments. For a given client project, we'd like to able to inject the implementation for S3 or filesystem persistence, but since attachment_fu totally hacks our objects to bits, we have to write some exotic and smelly code to configure a given model to do what we want.<br /><br />This is not the first time they've lodged this complaint, which is more or less that no matter how much (by much I mean well-defined, encapsulated, well designed OO etc) of an interface one of our in-house APIs surfaces, we are always already limited by the hackers who infiltrate our code with no contract. Plugins like attachment_fu offer no contract, they simply save the desperate ruby hacker from having to write lines of code, but the end result is a totally non-transferable hunk of smelly spaghetti code.<br /><br />Perhaps Rails just doesn't support our over-arching design goal, which is the ability to write one piece of code to handle the use cases all of our clients ask for on every project we do. Maybe it's faster to just re-write the same code over and over. I called this article "Rails has no application context" because I am used to working in an environment like Spring where I can request any implementation I want through the application context. Objects communicate through an interface and there is no shared knowledge of implementation details. I want this same kind of service bus in Rails, where objects can register themselves and be available, and other objects can request an implementation of an interface and know that the required operations (eg save to storage mechanism) will be available without having any idea about implementation details of each other. <a href="http://wiki.apidesign.org/wiki/Blogs">Jaroslav Tulach</a> calls this a "Teleinterface" which I think describes this situation accurately.<br /><br />I've proposed to my development team that we create some kind of proxy for attachment_fu to rape as the sacrifical vigrin, rather than treat our whole domain model as an open harem. We must repel the godless Romans from our code. We can create some kind of proxy object that gets wired up with the attachment_fu hooks, and this object can then act as an API for our code to consume.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6724296357282256647.post-752995729590132642009-03-31T16:04:00.001-04:002009-03-31T16:16:39.739-04:00The road to hellThe road to hell is paved with implementation detailsUnknownnoreply@blogger.com0tag:blogger.com,1999:blog-6724296357282256647.post-43793430599883484802009-03-30T11:06:00.005-04:002009-03-30T11:18:38.754-04:00Agile isn't easyI'm sure other people tranistioning to agile have had this thought. Isn't agile supposed to make my job easier? Won't I have more time to golf and relax? And yet, agile seems to be harder, and require more work and effort, than doing things business as usual. Well, guess what? Executing great work at the highest level is hard! It's harder than business as usual. But it pays dividends. You delight your customers, you have a happy, satisfied team doing valuable work.<br /><br />The moment the team stops doing work against stories its committed to, the magic leaves the room, quickly and dramatically. This is your fault as the product owner or business manager, not the teams. Business inputs will constantly distract you, customers will constantly derail you, personal issues will manifest in your team's lives. The world will never cease in its efforts to derail, and that's why you have the hardest job of all as the product owner. You're the glue holding the whole thing together, keeping the world at bay.<br /><br />It actually reminds me a lot of skeet shooting. Shotguns are weird weapons, and skeet shooting is quite the challenge: you're trying to hit a moving target with an explosive burst of shot. You're not sitting there at the range, lying prone, taking as much time as you need to put the bead on the bullseye. You've got at most 2-3 seconds where the clay will be close enough to the target area where your shot will hit it with any force.<br /><br />The magic of the shotgun is that you don't aim down a site. You line up your vision to the sites - you put your entire upper body and vision in alignment so that you can follow the target with your eye. You no longer aim. You see the target line up, you squeeze the trigger, and pow! it explodes.<br /><br />This reminds me a lot of setting the goals for a sprint. It's all mental. You have to line everything up to hit the target before shouting "pull!" and if you don't, you're guaranteed not to hit this elusive target. Go down to your local clay shooting range and try it out, you'll find it weirdly identical to leading a software team. Take your whole team as a team-building excercise. There's something interesting that happens when you fire deadly weapons together safely.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6724296357282256647.post-32768310308158766222009-03-25T17:30:00.000-04:002009-03-25T17:31:46.376-04:00Google Guicedon't know how i missed this... http://code.google.com/p/google-guice/. "Put simply, Guice alleviates the need for factories and the use of new in your Java code. Think of Guice's @Inject as the new new. You will still need to write factories in some cases, but your code will not depend directly on them. Your code will be easier to change, unit test and reuse in other contexts. "Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6724296357282256647.post-41065329994027164232009-03-25T00:36:00.002-04:002009-03-25T00:36:50.853-04:00Business ProcessI have to write an actual full post about good customers and business process - how are you going to go from unloading product on the loading dock to getting it to your customers' front door?Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6724296357282256647.post-65938470287301971752009-03-24T18:17:00.007-04:002009-03-25T01:39:01.071-04:00High quality developers and language flame war retardsHaving now built out my team of developers, I continually refer back to my experience of being interviewed by and performing interviews for Amazon. I've been feeling a lot of frustration at Ruby Kool Aid - and not for the usual reasons at all. The real annoyance for me is just like Republicans cloaking themselves with the flag while they systematically looted the country for 25 years and sent it to hell, Ruby foamers think they know how to write good software because they can bang out a RoR site quickly. The reality is that the Ruby scene has become densely populated with a large number of charlatans who are just like the sucky php and perl hackers from days of yore: array bashers and hash smashers who put 2000 lines of junk in a controller and call that agile high quality scalable code.<br /><br />I've had 2 major problems with software in the past: unmaintainable code and performance. Sooner or later, when enough people are pounding your website at the same time, you'll have performance problems, as any decisions you made that are suspect will get grossly magnified and you'll have to do some careful analysis and get it fixed. I don't care if you write in Java or C# or RoR, this will be true. But if you wrote good maintainable code that follows principals, it again doesn't matter what language you wrote it in; you can fix it. And the reality is, regardless of technology, most software gets thrown out at this stage due to the steaming shit pile condition it is usually in by the time performance becomes an issue. Just like any other solution, RoR doesn't and can't force the desperate hackers to write good code; but for those that know what they're doing, they can write well-designed software in any language or environment (well maybe except php hahahaha just kidding).<br /><br />This was the Amazon way of finding talent. You looked for people who could solve problems quickly, efficiently; who could handle the stress of a day of hard problems; who liked to bite their teeth into a challenge; etc. They were language agnostic in the interview. You could write any kind of pseudocode you wanted. It was all about well-formed thought under pressure. This of course was because the existing code base was so bad that you had better be able to fix super gnarly problems after getting paged at 3am but that's another story.<br /><br />I swear I will never become an evangelist for any type of technology. I will always promote developers delivering high business value software that achieves the objectives. I think if you are still in the software game in your 30's and beyond, you get bitter -- I mean ok everyone who makes it out of their 20's in one piece is going to be a little bitter -- and you kind of keep on fighting last year's war. I think a lot of java veterans who are now Ruby nuts do so because of spending years and years working on million-line J2EE projects from hell that were designed by management and not developers. Again, Ruby isn't the solution: the solution is letting engineers design software that delivers business value.<br /><br />The true war I fight and the true problem with the profession is the never-ending gap between business stakeholders and implementation teams. Both sides need to develop a shared understanding of business value and then go implement it. Now in Rails' favor, you can get a lot more done quickly to get something up and running in Rails, absolutely no doubt about it. The other day I was messing around with Apache Axis2 as I fantasized about replacing ActiveRecord with a bunch of Web service calls and a Java back-end, and I was immediately reminded of all the awesome things you don't have to deal with in Rails. There's quite a bit of fucking about just to get a java web service up and running. I think the challenge of desiging a well-formed API is just as hard in either language, but Rails lets you sketch it out a lot faster. Anyway I don't care how it gets done, in any environment, if you know what you're doing, you'll get 80% of the way there in a few days and then spend 3 weeks getting the last bits right.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6724296357282256647.post-88707292131105606982009-03-24T16:11:00.004-04:002009-03-25T01:39:41.211-04:00Not the only one who thinks constraints are required<a href="http://www.typepad.com/services/trackback/6a00e54fdca911883300e550107b318834">An older post</a> by Obie Fernandez speaks to the constraint thread I find running through many disciplines. I've now spent almost 8 months trying to bring agile to a traditional design agency and we've had some amazing success and terrible failures. I went back and read through all my blog entries I created at the start of this journey and I never explained what my thoughts and perspectives are on constraints.<br /><br />When solving a problem in a problem space with almost unlimited possible solutions, such as architecture, graphic design, or software engineering, the use of an over-arching design principal is absolutely required for success. Really good solutions often rendre the underlying principal invisible. The visitor to the solution site (words, websites, code) may not be able to immediately artiuclate what the principal or principals were, but they can immediately tell a well-ordered paradigm. The Kool-Aid crowd of the RoR scene has embraced a very strong design principal in Rails. What are some other examples of this principal at work?<br /><br />Graphic designers have to tell a story when creating their portfolio. There has to be an arc to the story they tell. It's often useful to think, ok, this portofolio is going to evolve along the color wheel. The start of the book will be red, then green, then blue. I will tell a story using color. The solution is transparent, yet the visitor feels a smooth sense of flow to the book without knowing why. It doesn't matter whether they identify the arc or not, its existence and the book's adhearance to the rule makes it coherant.<br /><br />Usability is the study of creating a coherant language in your software site. The use of personas and stories with a "so that" clause are great examples of creating a well-defined guiding principal that leads to a condition of self-contained coherance. It is very much a semoitic practice. Well designed software contains a very easy to grasp completely consistent internal logic. It may be for grandma instead of Skippy Jr., but it doesn't matter; when the logic is consistent (due of course to rigid adhearance to a simple constraint), the user is able to explore and map out the semoitic rules of the system. The surface of the software is discoverable and the way through the rabbit hole quickly becomes well-known.<br /><br />OOAD uses this same technique. A well designed object has multiple constraints: closed for modification but open for extension; clearly defined responsibility; encapsulation; loose coupling; etc. When you read code that follows these prinicipals, you always already know what it does when you read it. The more you read the good code out there, you start to absorb the zeitgeist of patterns subliminally, and you start to try to factor your own code to follow these well known best practices.<br /><br />Biggest take away from trying to do things right: It takes constant, ceaseless, undying devotion. The difference between doing right and doing wrong is not the amount of work or the intensity factor. It's just that you get a lot more high quality work done in less time. Desiging good OO code is really hard and coders under pressure will always revert to desperate perl hacking with arrays and hashes. Doing a stand up every dayis hard because something more "urgent" (but less important) will always come up. Tracking velocity is brutal because every day it tells you that you and/or your management (or client) is full of shit and you're not going to get it all done on time (probably because most of what you're doing is not valuable anyway and you haven't thrown out the junk with enough constraints - you were driving to get the dollar rather than being driven by a well defined ROI MMF business reality).<br /><br /><span style="font-weight: bold;">Biggest Failures</span><br /><ul><li>No code review, whether by pairing or old school diffing code check-ins</li><li>Letting ruby kool aid drinkers baffle me with bullshit when i should have seen the desperate perl hacker array and hash smash from miles away (badly named classes with no clearly defined responsibilities, more spaghetti than the tower of pisa, etc). </li></ul><br /><span style="font-weight: bold;">Biggest Successes</span><br /><ul><li>Creating a true build and deploy cycle so that bad code doesn't escape into the wild</li><li>Pivotal Tracker - it is a relentless pain to constantly drive a whole team, from business leader stakeholders down to developers to all communicate in terms of business value, but the moment you let the devs go off into implementation details or the business owners look for short cuts and quick cash, you will lose your guiding principals, all benefit and value of simple constraints and SDD goes out the window. You can't ever stray from the noble truths. </li></ul><br />Ok this post is like a mega brain fart that has books worth of ideas in it, but i just needed to blat some of this out and get on paper so to speak. I need to really develop this idea of design constraints as a critical component for success. My unique experiences in both the design and software fields, combined with my philosophy background, gives me an interesting perspective (or the ability to endlessly romance the sound of my own keyboard clacking, you decide).Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6724296357282256647.post-59996056974079983762008-08-31T22:20:00.002-04:002008-08-31T22:22:56.671-04:00The Cast of Characters You Will Meet in your Agile Customer Journey<ul><li>The derailer</li><li>The snake oil salesman</li><li>The bargainer</li><li>The visionary</li><li>The frustrated individual whose talent is being wasted</li><li>The fiefdom warrior</li><li>The lazy bugger who doesn't want to put any effort into bringing their business to life</li></ul>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6724296357282256647.post-41469833880710469792008-06-05T09:14:00.003-04:002008-06-05T09:19:52.669-04:00What books to read and why to formulate your own methodMy challenge is to create a planning and tracking system for an agency with many customers and projects in varying states, on-going support and maintenance issues, and fixed deadlines. Here are some observations:<br /><br /><ol><li>Dates are fixed, but projects appear to be 1-3 months behind</li><li>Different types of work are not timeboxed - there are no boundary conditions on work activities to create focus</li><li>As a small company, people have to wear many hats, but some hats work better than others. This has meant that planning has taken a back seat to selling and doing, which makes sense in this environment.</li></ol><br />In creating a system, I've suggested reading:<br /><br />Agile Project Management With Scrum - some of the ideas don't really carry in the fixed-date scenario, but this book does a great job of showing the effectiveness of boundary conditions, the requirement for creativity and improvisation in real life (there is no silver bullet), and the value of empiricism: observe what is happening and make new decisions based on this reality.<br /><br /><br />Agile Estimating and Planning - the most useful hands-on guide<br /><br />Extreme Programming Explained - embrace change and set yourself up to successfully live with and manage changeUnknownnoreply@blogger.com0tag:blogger.com,1999:blog-6724296357282256647.post-72917341986532036172008-06-03T23:46:00.001-04:002008-06-03T23:48:18.130-04:00Storyboard ArtDudes are hardcore about their planning/tracking boards.<br /><a href="http://www.flickr.com/photos/agileinaction/"><br />http://www.flickr.com/photos/agileinaction/</a><br /><a href="http://www.think-box.co.uk/blog/2008/05/simplified-planning-board.html">http://www.think-box.co.uk/blog/2008/05/simplified-planning-board.html</a>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6724296357282256647.post-21226957517124072692008-05-30T16:41:00.003-04:002008-05-30T16:50:20.574-04:00It's not the shoes, Mars...Great article on the idea of <a href="http://www.infoq.com/news/2008/02/personal-agility">personal agility</a>. People on a team need to feel a personal sense of responsibility to their projects. More interestingly, they need a certain amount of emotional intelligence and personal development to be effective leaders. Traditional carrots and sticks as we know only work for some people.<br /><br />The best performers at my last job were people who had a carrot bigger than the paycheck, e.g. they needed a green card, or people who just loved the thrill of chasing down the solution to the difficult problem, regardless of the impact this problem might be having. Anyone who did not have these characteristics generally failed to get promoted on my last team.<br /><br />It looked like they only promoted people who worked 60-80+ hours a week, but it's interesting to think about why those people were willing to work those hours on problems that were the result of what I would characterize as management shortcomings, when the majority of the team was not. It is because the majority of the team had no incentive to work on things they knew were dumb, things that had a small impact on customers, or things that were just painful to work on.<br /><br />In other words, personal agility is not a new way of commanding and controlling. It is a paradigmatic shift where management must lead by example, show and not tell, and instilling a sense of ownership (and developing a consequent sense of leadership). This is not something you can do by brute force, it is, like house music, something you need to feel. "If you don't feel the stirring in your heart after the first few bars of a tango, go find something else to do". (Rough paraphrase from 'La Cafe de los Mastros').Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6724296357282256647.post-85994496891465129312008-05-27T23:15:00.003-04:002008-05-27T23:26:33.593-04:00Design for Today's Problems, Not Tomorrow'sp. 110 of Extreme Programming.<br /><br /> "The traditional strategy for reducing the cost of software over time is to reduce the probability and cost of rework. XP goes exactly backwards. Rather than reduce the frequency of rework, XP revels in rework...<br /> "The key is that risk is money just as much as time is money. If you put in a design feature today and you use it tomorrow, you win,because you pay less to put it in today.... However... If there is enough uncertainty, the value of the option of waiting is high enough that you would be better off waiting.<br /> "Design isn't free. another aspect of this situation is that when you put more design in today, you increase the overhead of the system. There is more to test, more to understand, more to explain. So every day you don't just pay interest on the money you spent, you also pay a little design tax. With this in mind, the difference in today's investment and tomorrow's investment can be much greater and it is still a good idea to design tomorrow for tomorrow's problems.<br /> "As if this weren't enough, the killer is risk.... You can't just evaluate the cost of something that happens tomorrow. You also have to evaluate the probability of it happening....<br /> "<span style="font-style: italic;">So, the cost of making a design decision today is the cost of making the decision plus the interest we pay on that money plus the inertia it adds to the system </span>[emph. added]. The benefit of making a design decision today is the expected value of the decision being profitably used in the future.<br /> "If the cost of today's decision is high, and the probability of its being right is low, and the probability of knowing a better way tomorrow is high, and the cost of putting in the design tomorrow is low, then we can conclude that we should never make a design decision today if we don't need it today." <br /><br />Example: From Chapter 3<br /> "Suppose you're programming merrily along and you see that you could add a feature that would cost you $10. you figure the ROI on this feature (its net present value or NPV) is somewhere around $15. So the NPV of adding this feature is $5.<br /> "Suppose you knew in your heart that it wasn't clear at all how much this new feature would be worth -- it was just your guess, not something you really knew was worth $15 to the customer. In fact, you figure that its value to the customer could vary as much as 100% from your estimate. Suppose further that it would still cost you about $10 to add that feature one year from now.<br /> "What would be the value of the strategy of just waiting, of not implementing the feature now? Well, at the usual interest rates of about 5%, the options theory calculator cranks out a value of $7.87.<br /> "The option of waiting is worth <span style="font-style: italic;">more</span> than the value (NPV = $5) of investing <span style="font-style: italic;">now</span> to add the feature. Why? With that much uncertainty, the feature certainly might be much more valuable to the customer, in which case you're no worse off waiting than you would have been by implementing it now. Or it could be worth zilch--in which case you've saved the trouble of a worthless exercise.<br /> "In the jargon of trading, options 'eliminate downside risk'".Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6724296357282256647.post-55312684901774990172008-05-27T22:20:00.002-04:002008-05-27T22:22:36.794-04:00Plan for priorities, not developmentFrom <span style="font-style: italic;">Extreme Programming Explained: </span>"Planning for priorities vs. planning for development -- Ber aware of the purposes of planning. Planning so the customer can establish priorities needs much less detail to be valuable than planning for implementation, where you need specific test cases."Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6724296357282256647.post-58419738867854865132008-05-22T22:24:00.001-04:002008-05-22T22:25:36.627-04:00Simple Story Template<a href="http://blog.mountaingoatsoftware.com/?p=24">Great post</a> on a simple story template from Mike Cohn. "As a <foo>, I want <bar> so that <baz>". Simple constraint.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6724296357282256647.post-63063768098865418482008-05-17T23:17:00.000-04:002008-05-17T23:20:34.579-04:00Scrum requires course correction at every turnScrum is a process of correction. You don't just start doing it and then everything just magically works. The first Sprint will have failures, processes that didn't work, procedure that wasn't followed, ideas that didn't sink in. At the end of the Sprint you review and course correct. Individuals on the team constantly strive to hold up their end of the Scrum contract, while the whole teams looks for ways to change, adapt, evolve, and innovate. It's just like sailing, there's constant course correction that goes from coarse to fine as you get closer to port.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6724296357282256647.post-24863750000514436392008-05-17T19:16:00.000-04:002008-05-17T19:19:28.728-04:00Scrum requires teamsIn order for teams to manage themselves, they need to be dependent on one another for direction. They can not look to the ScrumMaster or Product Owner for direction or orders. The team uses its abilities ranging from subject matter expertise to best gut instincts to make its own decisions. It is very much like a sports team where everybody has a role, but the team wins games as a team, not as individuals. In Hockey, you get the same amount of points for an assist as you do for scoring a goal and it is the same in work. <br /><br />The reason Scrum and stand-ups never worked at my old job was because "team" members were pitted against one another. It was a Lord of the Flies, survival of the fittest, where progress was always reported to some kind of boss man. No one in the group ever had a stake in the success or failure of the work other team members were doing. It was much more like a World Rally Cross event, where team members all drove Subarus but were conducting their own race, competing with each other to win each stage of the rally. It was awful.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6724296357282256647.post-31049568508316696272008-05-17T15:01:00.000-04:002008-05-17T15:04:07.955-04:00Why is this here?This is the question we must always ask ourselves - why am I doing this now, why is this artifact here, etc? It's more efficient to spend some time to analyze what the next important task in our project is and define the point on the horizon towards which all activity and thought will converge for a given time-limited iteration will converge. This is the freakin management, development, and creative zeitgeist of the late 20th century. Agile development, Scrum, "Getthing Things Done", the E-Myth, it's all saying the same thing: get out of your own way and focus. Use the power of disciplinary thinking to your advantage - it is effective.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6724296357282256647.post-59558088398162497782008-05-17T14:58:00.000-04:002008-05-17T15:01:41.901-04:00Story Driven DevelopmentEffective thinking about complex problems means bracketing off. SDD is one way of doing that.<br /><br /><ul><li><a href="http://www.easyb.org/">EasyB</a></li><li><a href="http://www.scottberkun.com/blog/2007/asshole-driven-development/">Amusing compendium of anti-patterns of development (Asshole Driven Development)</a></li><li><a href="http://dannorth.net/whats-in-a-story">A very good intro to behaviour/story-driven development</a></li></ul>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6724296357282256647.post-84625560902913372862008-05-17T14:47:00.000-04:002008-05-17T14:58:32.216-04:00Scaling RoR when you're a Java developerSo I'm coming into a new situation and I can't afford to make dumb decisions based on my own experience and bias. The company has made a large and profitable investment in doing smallish websites using RoR. Problem: maintaining 50+ individually deployed websites (regardless of tech stack) is a drag, so I'm looking for ways to consolidate onto a single platform that can host multiple customer sites. <br /><br />My hunch as I go into what a former co-worker described as "hardware is expensive-land" is that RoR scales via a "throw money at the train" algorithm. I'd like to be able to scale out layer by layer. Blah blah blah the point is, I am a java developer and there are more well-known patterns for scaling out large java web apps then there are for ruby. At a small company we need proven solutions that a) work and b) aren't too hard to do. The possible trajectories and migration paths at this time I see are:<br /><br /><ol><li>JRuby</li><li>Grails</li></ol>JRuby means I can dump RoR apps into a Java container and get some free clustering. More interestingly, I can transparently migrate RoR controllers onto Hibernate without having to refactor much - a testament to the greatness of the RoR separation of concerns.<br /><br />Grails means I can move to an all Java platform using a very RoR-like idiom that will make life easier for the developers. <a href="http://headius.blogspot.com/2008/05/power-of-jvm.html">Groovy 1.6 looks to have the performance problems of Groovy</a> in the bag. It uses Hibernate, the one technology I feel like I have some strong knowledge of, strong intermediate if I were to use skier analogies.<br /><br />Grails/Groovy books<br /><br /><a href="http://www.apress.com/book/view/1590599950">Definitive Guide to Grails, Second Edition</a><br /><a href="http://www.pragprog.com/titles/sdgrvr/groovy-recipes">Groovy Recipes</a>Unknownnoreply@blogger.com0