Overview

Over the last 8 months I’ve been swamped working on a very interesting project. Between this project and a set of other commitments, I’ve literally spent 16h a day working. The blog has been neglected, but I plan to take up writing again. I’d thought I’d take some time to reflect on the project, lessons learned, etc.
This first installment is really just a quick overview of the project.
The project has been a learning experience for me. My plan is to share some of these insights with you all in upcoming blogs. In particular, I’d like to discuss:

• The experience using a DSL to ensure tight integration between software groups
• The relevance of Shannon-Hartley’s theorem on development with DSL
• The challenges integrating User Experience into agile development processes
• One approach to distributed agile development

The Project in Numbers

First, let me describe the project (without getting into too much details, as the project is yet to be publically launched). Here is the project in numbers:

• ~ 1000 classes
• ~300 pages (~20 of the pages are Web 2.0 heavy, almost complete applications in their own right)
• ~8500 artifacts (source files, xml configuration files, view files, etc)
• 100+ uniquely designed page flows
• ~70 distinct different persistent entities
• Complex analytics processing very large data sets
• Small development team (3 on the back end, 2 on the front end)
• Large analysis team (10-15, which may seem an overkill, but the software is carving a new business segment and hence defining the requirements is as much of a design as it is a capture)

More about the project

Although I’ve been attached to the project for 7 months, the actual development of the project did not start before late January (or about 4 months ago).

The project management was agile (with modifications that I’ll discuss later) and the requirements where in constant flux.

As far as techniques and technologies go, here are the highlights:

• Technology:
• Java and JEE Spring framework
• Hibernate
• JSP
• jQuery
• Tools
• Sococo
• PivotalTracker
• Eclipse
• Maven
• Toad
• Spring Roo
• A few Domain Specific Languages (DSL’s) from SciSpike

Lessons Learned

I’ll write up more details later, but here is in short what we learned.

DSL’s work

At SciSpike, we have been convinced of this for some time and this project did not change our opinion. We have collected metrics from other organizations and the production of the team is a factor more productive than even the best metrics we’ve collected.

Formalization of contracts using DSL’s is essential

The project had one team working on the front end and another team working on the back end. The teams were located in different cities and the potential for integration problem was a huge risk factor. We decided to define a contract language that formally specifies the interactions between the front and the back end. This language has been essential to minimize the integration errors.

Domain Model First!

The idea (often pushed by militant agile proponents) of iterating yourself to a correct solution by building feature by feature is suboptimal. You MUST ensure that you have an overall plan of what you try to do. If this is too waterfall-ish for some, so be it. We tried both approaches, and there is no doubt in my mind that the more sequential approach better.

DSL’s can make you go too fast

There is a danger of speed blindness when using DSLs. When using a more traditional approach, you have much more time to think about your decisions (during detailed design, coding, testing, etc.).With the DSL’s, we felt like we spent an hour or so thinking about a topic intensely, then generate a solution for then go on to the next topic. Although this sounds great, it adds some potential risk. The team does not spend sufficient time letting the design segment. I’ll try to write up a convincing blog entry on this later, because I know this probably sounds absurd (it certainly took me by surprise).

Don't go blindly into that Agile promise

I teach agile, I believe in agile, I've had success with agile. When I teach and when I practice, I have a template I'll start off with. It will get the project pointed in the right direction, however, the process may be modified at any time. Adapt the process to the team and the environment.

Coming up...

My plan is to tackle the lessons learned one lesson at a time in upcoming blogs. The project will have its official launch soon and I think it would be best to wait until I can exemplify with real screenshots and code examples. Stay tuned...