Petter's Random Thoughts on Software

Debugging Trick in Eclipse

2011-09-02T15:13:00.001-07:00

Introduction

Today a friend asked me if you could execute an action in Eclipse on a breakpoint. I thought for sure that that would work, but decided I had to check before I said yes. Well… there is no such feature in Eclipse, but here is a cool work-around.

Problem

The problem that my friend had was that he’d like to inject some code when the breakpoint triggered, print out some stuff, then resume operation as quickly as possible. The reasons for this may be many, but the most typical is that you’re running in some real-time (or semi-real-time) scenario where stopping in the debugger may make you miss a deadline.

Solution

Create a specialized break in Eclipse that has the following traits:

Make the breakpoint conditional
Specify that that you only want to suspend the thread when the condition is true.
Create a condition with side-effect

Example

In this example we’ll show some simple code that simply iterate through all numbers from 0 through 99 and summarize them all. At the line where we accumulate the sequence, we’d like to print out the number and as quickly as possible continue the execution of the program.

Here is some the sample code:

   1:  package com.scispike.demo;

2:

   3:  /**

   4:   * Dummy class created to

   5:   * demonstrate some debug tricks

   6:   *

   7:   * @author pgraff

   8:   *

   9:   */

  10:  public class DebugDemo {

11:

  12:      public static void main(String[] args) {

  13:          int j = 0;

  14:          for (int i = 0; i < 100; i++)

  15:          {

  16:              j += i;

  17:          }

  18:          System.out.println(j);

19:

  20:      }

21:

  22:  }

The example is rather silly, but it will do for this purpose.

Steps to make the print work…

Create a breakpoint on line 16 ( “j += i;”)
Go to the debug view in Eclipse
Right-click on the breakpoint and select properties
Enter the code to print out ‘i’ and return false (as shown below):
Run the program in debug mode

Notice that the application now print out the i on all iterations!

Anecdote

I was quite exciting when I saw this. If you are familiar with Aspect Oriented Programming, this is quick way to patch in an aspect! Here is another silly example.

We are now patching the value of ‘j’…. I just can’t wait until someone leaves their computer on while they debug a problem…. Can you imagine the look on their face as I make some random assignments :)

Summary

In this article I shown an advanced debugging technique not commonly known in Eclipse. The technique can be used to quickly print or patch values without slowing down a program significantly (or have a lot of fun with your colleagues).

On Performance

2011-06-02T15:29:00.001-07:00

Description

Over the last 25 years or so, I’ve been often tasked to help optimize the performance of various solutions. I keep careful records of all the work I do and I recently decided it would be a good idea to collect all my thoughts/experiences around performance.

I’m actually not only collecting notes on performance, rather I’m trying to categorize all my notes. Unfortunately, most of my notes are in old notebooks and they are filling up several shelves at my house. I’d like to get some of the notes synchronized and made electronic so that I can make use of them in my consulting/teaching. I expect this will take many years…

One idea that I had was to organize the material around architectural viewpoints based on views that I’ve developed over the years (primarily using the IEEE 1471-2000 framework. For some of the viewpoints I literally have 10’s of thousands of notes, so I thought I’d tackle some of the smaller viewpoints first. So… for my next x blog entries, I’ll extract notes on performance.

To make these notes useful to others, it would probably be best if I discussed some of the topics that I take for granted. These are issues that my notes will not discuss because they are principles I consider them given (or immutable if you’d like).

What influences performance?

In my experience, there are only a few things that are worth focusing on:

Algorithms and data structures. E.g.,

Simple algorithmic improvements:

Bubble-sort versus quick-sort
Graph-search algorithms

SQL query optimization

Indexes
Proper use of the database expression power (SQL)

Locality of reference

Caching
Memory usage
Taking advantage of immutable knowledge

Processing power

Scaling by adding hardware or reassigning resources

Process of performance engineering

Make sure you understand what are desired performance characteristics of the system you are trying to optimize. Sometimes the total through-put is most important, sometimes its reactiveness is most important, sometimes various scenarios have very different performance requirements. In other words, make sure you understand the problem before you start optimizing.

One of the most common errors I’ve seen is the premature and blind performance optimization. In other words don’t optimize before you know where you have a problem. Some performance optimization techniques come at a high expense from other architectural perspective (such as reuse, maintainability, continuity, …).

By premature optimization I mean performance optimization that are suggested before we even know we have a problem. Blind statements made by someone in the organization that are blindly applied across the organization. E.g., using getters provide a function overhead and hence should never be used. Let’s make the data members accessible to the client. Such a decision (apart from being wrong in many environments) has a high cost and are often not very effective.

By blind performance optimization I mean: DON’T optimize without profiling the application. More often than not, the ones optimizing are optimizing the wrong thing (e.g, who cares if you can save 1 millisecond by reducing a method call if the problem is that you are using several seconds in some database query…).

Let me illustrate the last point with an example (an old note from my notebook). Many years ago I was working on a large scale distributed real-time system. The system we were working on had an extensive framework on which we build a large set of specialized applications for various clients. The system had serious performance problems. When an operator pressed a navigation button to move to another view, it took seconds to move to the next screen.

The framework team were called in to an all-hands to improve the performance. The performance improvement went on for many months and the engineers were optimizing based on a set of principles set forth by one of the most experienced engineers in the company. Even so, the progress was slow. After about 9 months, they managed to optimize the performance with ~ 20%. Some of the performance optimizations degraded the architectural structure (often you can steel CPU cycles by collapsing architectural layers).

My team was eventually also called in (we were working on applications, but we had a good reputation and the framework team was getting desperate). Debugging the application was rather cumbersome (at that time we were using an in-circuit emulator (ICE) that very few engineers in the organization mastered. On my team, one of the engineers had been working on the design of the ICE and knew it inside and out. Using the ICE and a profiling tool, he and I sat down and started to narrow down the areas where we spent time. This was probably the first time this problem had been attacked this way. To make a long story short, after a few hours, we had optimized the performance by 40 times. A day later, the system ran 250 times faster. After a week the system ran many thousand times faster.

How did we do that? It turned out the problem was in a few lines of code. It was an algorithmic problem (unnecessary multiplication in a loop in the display driver). There was nothing we did that the other engineers would not be able to see, nothing particularly clever, no magic. It’s more like finding your car in a large parking garage… but we ‘cheated’ by turning the lights on.

Summary

This is the introductory article where I’ll be focusing on performance optimization. The idea is to collect some of my notes from my work since leaving school. This first article was written to establish some basic principles that I will assume known in the articles to follow. Some of these principles are:

Don’t optimize without understanding the performance goals
Don’t optimize without a profiler (or some way of measuring performance)
Performance comes foremost from algorithms, locality of reference and computing power

Why DSL’s are such a hard sell

2011-04-06T09:19:00.001-07:00

Overview

Over the last years I’ve built a dozen or so large scale Domain Specific Languages (DSL) and almost without exceptions, they’ve been hard sells. Usually, each of these received initial resistance in the organizations they’ve been introduced. This despite their convincing potential and their eventual success.

Here are some examples of these successes:

Project for Company X. Created a set of DSL’s for various aspects of their development. Two main effects:

Average time to develop an application went from 4 man-years to 3 man-weeks (factor of 70 improvement)

Project for State Agency Y. Create a DSL to handle a complex connection between implementation layers

Average time to develop a connection went from ~one week to 10 minutes (factor of 240 improvement)

Project for Company Z. Created a DSL to generate variations of application that track various initiatives (metrics not in yet, but we expect a similar improvement in development effort as in the other projects on this page)

In this blog post I’ll focus on the kind of resistance one may expect and why.

Building our own language, You’re kidding right?

The most common objection is to the presumed effort required to create a DSL. Few have experience writing a language and those that do typically wrote languages some time ago.

Fact is, if we suggested that one build a language to solve their problem a decade ago, it would have been a very ambitious project. We would have looked at a massive project taking months if not years to complete.

With today’s tools though, writing a DSL is what I usually categorize as a “weekend project”. Although there have been languages that have taken me months to complete, the typical effort is in days (or worst case weeks).

The latest tools in the open source and also in the commercial world are very different from the ones available say a decade ago. Here is a typical estimate for a language…

Design of the language (the abstract syntax) – Typically 1-3 days
Implementation of the concrete syntax – Typically 1 day

Including editors with syntax highlighting, auto completion, …

Transformers – 1 day to weeks…

This is often the tedious part where you generate code or other artifacts from the abstract syntax

OK… the times above are not true for everyone. It requires that you have intimate knowledge of the language generation tools, but achievable I believe for any reasonably skilled engineer.

My way around the concern of effort is BUILD THE TOOL FIRST, THEN SUGGEST IT. Because the effort is rather moderate, I can usually build a first cut of a tool over a long weekend or while flying around (no kidding, I do my best work on airplanes :)). The realization that a language can be created over such a short time and already be useful typically removes the concern of complexity.

The approach you suggest sounds a lot like a 4gl. We tried that before and it was a disaster

A 4GL and a DSL are different things. A 4GL was a general purpose language at some high level of abstraction (I have nothing against that by the way). A DSL is a special purpose language preferably at the HIGHEST level of abstraction possible. Believe me, they are different.

Again the remedy is education, but the association between experiences on 4GL’s and expectations of a DSL is real and something you must prepare for.

That is going to be a disaster for recruiting… Who would want to work on that?

Ouch… so true… and so hard to argue against… I’ve tried a myriad of ways to work around this. Let me give you a few examples.

The Parallel Approach

The parallel approach means that we create a DSL that generate code, but also allow for the developers to do the same by hand.

This has the following effect:

We are still using the general purpose language… It is just that you can use this tool to makes you more productive.
Eventually, the DSL becomes the dominant or the only approach

The Hook Approach

Leave something to be done in the general purpose language. Don’t hide it all. This usually make a lot of sense anyway. There is often some part of the the problem that may be best solved by a general purpose language. Typically, this is some procedural part of the problem. As long as SOME PART is written in the ‘recruiting language’ the solution may be a lot more palpable.

I’m hoping that some of these objections will disappear with time, but right now they are real and I have to admit that I’ve modified DSL’s ensure that programmers get to program in the general purpose language, EVEN THOUGH I KNOW HOW TO GENERATE THAT CODE TOO and it would have been more productive.

We make money by the hour… Why?

I’m sure some of you think I’m kidding, but believe me! This is a common response from some development domains (typically government or contractors for government projects). This is the one argument that will usually make me give up head for the door. Personally, I don’t understand how organizations can live by those goals and openly talk about them.

Government! Change your metrics!!!

Summary

When you suggest the use of DSL'’s in software organizations, expect resistance. Very often this resistance is based on unjustifiable concerns and often the way around it is build it first… This eliminates the fear of the complexity of building a language, it also gives the stakeholders a chance to see what it would look like.

My favorite tool to show off the ideas are slightly different based on what kind of concrete syntax I want. If I want a graphical syntax, I prefer to use the Microsoft DSL tool (although, the graphical modeling framework of Eclipse (GMF) also works well). If I’m creating a textual syntax, my favorite tool right now is XText (Eclipse based tool).

iPad2 vs. Xoom

2011-04-03T02:29:00.001-07:00

I've been eyeing the new iPad, but when I heard about the delay (and I have to admit also my current trip to Asia and the thought of not having something to play with on the plane), I decided to try out the Motorola Xoom instead.
I've been using the old iPad and briefly tested out iPad2. I've never been all that imressed by Motorola's design, so my expectation was not very high. However, after 3 days of use, I'm sold. In this blog I'll give you some reasons why.
Reason 1: It's Android!
It is so nice again to be on a platform that is not controlled by 1 company (person). I can now write my own apps without having to get it approved by the almighty fruit.
Reason 2: Flash!
Need I say more? I heard prior to buying the device that flash didn't work, but there must have been an upgrade between when I read the article and my purchase. Flash is working perfectly. This allows me to see the learning modules we have without opening my Laptop!
Reason 3: Multitasking
True multitasking... Really cool.
Reason 3: Apps
When I read previous comparions of iPad and Xoom, I saw that one of the issues people have with Xoom is the lack of apps. So far I don't see why. The selection of apps are great. All the apps (-1, a big one, Netflix) works as good or better on the Xoom!
Reason 4: The screen
It just seems sharper than the iPad..

Published with Blogger-droid v1.6.8

Retrospectiva

2011-03-01T08:37:00.000-08:00

This post is just a quick update on the status on alternatives to the PivotalTracker.
I've been playing a bit with Retrospectiva. It seems to be a good alternative to PivotalTracker.
At SciSpike, we'll setup a free service for our customers to move their projects to Retrospectiva. The projects that we'll still involved with, we'll move to Retrospectiva before the free-period ends.
My plan is to create a converter that can take the data from existing PivotalTracker projects and move them to Retrospectiva with minimal loss.
I know that some of you probably want to continue to use PivotalTracker and I applaud that decision. I've always liked PivotalTracker and still think it is a great tool (although, overpriced)!
When I get some more experience using Retrospectiva for some real projects, I'll give you all an update of how they compare and maybe also some recommended agile practices in Retrospectiva.

PivotalTracker is going Commercial

2011-01-20T14:32:00.000-08:00

Ouch... I just learned that PivotalTracker will start charging for their online project tracking tool.

I have mixed feelings. On one hand, I recognize their right to make money on what I consider to be an excellent tool, but on the other hand, I feel tricked...

I have a bunch of projects that I'm tracking in PivotalTracker and been pushing it where ever I've been consulting. The pricing model is a complete surprise also. a) it is too expensive, b) it doesn't fit the way I've been using PivotalTracker.

Because I've been consulting in many places, I have many collaborators and the price that I would have to pay may be as much as $3600 per year! That's a cost that I had not planned for.

I do apologize to my customers that may have started to use PivotalTracker under the wrong assumptions. I will make an investigation of alternative tools and post it on my blog as soon as I get some time to do the investigation (probably mid February as my schedule is ridiculous at this time).

Anyway, I'll be back with another post with alternatives as soon as I get around to it. Maybe I'll get some people together and build an open source alternative? Please let me know if you're interested.

Function Objects in in Java

2010-11-13T22:52:00.001-08:00

This blog entry is really just a support entry for a YouTube movie I created today.

A few weeks ago, I taught a course on plain old Java… The students had LISP background and raised the questions:

Does Java have Closures?

Well, it doesn't really have direct support for closures directly, but we can achieve most of the design advantages that closures provide. In this short demo, I show how one may implement closures in Java.

Code Listing

Here is the final code listing for what I showed in the demo.

ClosureDemo.java

   1:  package com.scispike.demo;

2:

   3:  public class ClosureDemo {

4:

   5:      public static void main(String[] args) {

   6:          System.out.println("Gauss should have said     : " + sum(1, 100, new EchoFunction()));

   7:          System.out.println("Sum of squares from 1 to 10: " + sum(1, 10, new SquareFunction()));

   8:      }

   9:      public static int sum(int min, int max, Function f) {

  10:          int sum = 0;

  11:          for( int i = min; i <= max; i++)

  12:              sum += f.apply(i);

  13:          return sum;

  14:      }

  15:  }

Function.java

   1:  package com.scispike.demo;

2:

   3:  public interface Function {

4:

   5:      int apply(int i);

6:

   7:  }

SquareFunction.java

   1:  package com.scispike.demo;

2:

   3:  public final class SquareFunction implements Function {

   4:      @Override

   5:      public int apply(int i) {

   6:          return i*i;

   7:      }

   8:  }

EchoFunction.java

   1:  package com.scispike.demo;

2:

   3:  public final class EchoFunction implements Function {

   4:      @Override

   5:      public int apply(int i) {

   6:          return i;

   7:      }

   8:  }

Adobe Captivate 5–A Superficial Review

2010-11-06T18:54:00.001-07:00

Introduction

For more than 2 years now I’ve been using Adobe’s tools for creating e-learning material. The two tools I’ve been using the most are:

Adobe Captivate
Adobe Presenter

I’ve been stuck on an older version of Captivate for a while (Captivate 3). Although Captivate 3 has worked well for me, there has been one nagging issue. When I create movies in Captivate 3, each demo consists of a main flash file and a set of smaller flash files that captures the parts of the demo that requires full movie capture (e.g.., drag-n-drop). It turns out that when you that this is a problem when producing SCORM modules. This content ends up being presented as a white screen. We’ve also noticed quite a bit of lag issues.

I was excited when I heard Captivate 5 had been released and after having confirmed that it solved my issues, I bought it immediately.

This blog is about my experiences with Captivate 5 which as we’ll see have not been all good. I don’t want my blog entry to overly negative. Captivate 5 is a great tool, but some of the problems in it are so severe that I think they deserve some light.

Problem 1: It Doesn’t Work with Presenter!

I had previously produced a set of material for a company selling testing tools. Their clients had noticed the problem of Captivate 3 of lag and white screens. Hence, they were excited to hear that I had acquired the new Captivate and started to update my material.

I open up the Captivate 3 project and Captivate 5 detected the material as Captivate 3 and started the conversion to Captivate 5. I noticed how the tool now is much faster and the interface a bit better, but other than that, everything looked as normal.

Next I replaced the old flash files in Adobe Presenter with the newly published flash demos. Here is my first surprise…. The tool warned

THIS CONTENT MAY NOT PLAY IN PRESENTER

Really??? They are just flash files. I realized that maybe there is also a newer version of Adobe Presenter… So I updated Adobe Presenter and after that the problem went away. Great!

Then I reimported the flash demos into Presenter. This time, no complaints. I generated the SCORM modules from presenter, again everything looked normal. I continued doing the same work for a set of my other presentations and after many hours of work, I had completed the transformation for my client.

I pushed it over to the Learning Management Site (LMS) and then tried it out. Now I was in for a surprise. All the movies played in 5 seconds (no matter if they were 15 minutes long).

After a lot of investigation I found out that Adobe Presenter does not support Action Script 3. This is to me quite incredible oversight from Adobe. It seems as if the ones creating Presenter do not talk to the ones creating Captivate. Although marketing seems to have no problems claiming (from Adobe FAQ):

Does Adobe Captivate 5 integrate with Adobe Connect™ and Adobe Presenter?

It does. Adobe Captivate 5 provides tight integration with Adobe Connect software, allowing authors to improve training effectiveness by adding a layer of interactivity to Adobe Presenter presentations and Adobe Connect online meetings. Users can also publish Adobe Captivate files directly to Adobe Connect Enterprise Server, to deliver content through standard web browsers, and track and manage courses and learners

I contacted the Adobe Technical Support and reported it as a bug. After some back and forth (it seemed at first the tech support team was also unaware of this). I got this message back:

Captivate 5 swf files are AS3 files which are not supported by Presenter. Presenter only supports AS2 swf so an AS3 swf may or may not work properly. We have this issue on our radar , please standby for the updates.

So I’m waiting for the updates…. I’ve been waiting for a while now, but I have a feeling this is not all that high on Adobe’s priority list.

Problem 2: You better have a Fast Computer!

I do have some pretty good computers. My desktop has 2 dual core processors running hyper-threading (appears like 8 CPU’s). It has 16 GB of memory and runs 64 bit operating system. Great machine.

My laptop is also state of the art. It is a Dell 6400 with a quad-core CPU and 8 GB of RAM. When I bought it (about a year ago) it was the fastest laptop I could buy. Also a really good machine. It’s also running 64 bit OS.

My preference is to use my laptop to produce material. It is slightly more quiet and my microphone picks up a bit of background noise. I decided I would put some demos up on YouTube (see previous blog entry). To do so, you have to publish your content as Flash Movies (fla-files). This is one of the options when publishing content from Captivate, so no problem. I produced a few files without incident.

I had a demo that is 11 minutes long. I noticed that after about 6 minutes of production (Captivate plays the demo while producing the fla file) my machine became non-responsive. I had to finally kill Captivate and try again. Again, after about 6-7 minutes I had the same problem. This time I had to restart my machine. Now suspecting a bug in Captivate, I decided to kill all programs but Captivate while producing my movie. Again, the same problem, however, it produced some content. However, when I played the content afterwards, the voice and pictures where out of sync (by minutes!).

I now moved the file to my monster desktop and try to produce the movie there. This worked! However, while producing the fla-file, the CPU utilization was hovering around 60%.

Although I understand that the transformation of flash to video is algorithmically challenging, it seems only a few people I know have computers fast enough to produce the movies.

Problem 3: No HD Movie?

When demo movies to YouTube, it is quite important to build high-resolution movies. Most demos I see are so fuzzy I can’t really see what’s going on on the screen. I did a little research and found that the best format for YouTube is 1280 X 720.

This makes a lot of sense if you know something about PAL and resolutions for HD movies.

I decided to record one of my presentation’s in this format in Captivate. This worked perfectly. I created a flash presentation in1280 by 720. After having published it as flash (to our web site), I decided to also push it to YouTube.

To make a long story short, it turns out that Captivate can not publish fla files in this format! Their maximum with is 1024 (this is only for movies, not for flash).

This again seems like a major oversight from Adobe.

Conclusion

Adobe has always been one of my favorite software companies. I used to love Framemaker, I love Photoshop, Captivate 3 was revolutionary and Presenter worked well with Captivate 3. My recent experience with their software has been disappointing. The release of Captivate 5 seems premature and its lack of integration with Presenter quite puzzling. It also seems strange that they do not seem to have tested their tool for typical scenarios (e.g., how well does the tool work for publishing content to YouTube?).

I will keep my faith in Adobe though. I hope they will fix their problems soon.

I think I'll write an article later about IEEE 1471-2000... and the current state of the Adobe tools seems such great example of why you need to evaluate your architecture from multiple view points :)

Developing a Web Service in Eclipse

2010-11-05T18:07:00.001-07:00

I’ve been teaching a few classes on Web Services lately and my students have complained that there are no good description on the net for how to get started building web services on Eclipse. I’m sure there are some, but I was encouraged to create one myself, so here it is. I also wanted to try out Captivate 5 properly, so most of this tutorial/blog will involve videos found on YouTube and separately on our e-learning site.

I’ll basically take you through the following two steps:

How to download and install the required tools?
How do I configure Eclipse to develop and run web services?
How do develop a simple Web Service?
How to deploy your Web Service?
How to test your Web Service?

I decided to use videos for this… So, each section is a separate little demo in the form of a Flash movie (sorry iPhone users… I may put the movies on YouTube later to make sure you can see them too).

Downloading and installing the required tools

The first thing we need to do is to download and install the tools required. The tools I’m going to use are:

Apache Tomcat
Apache CXF
Eclipse JEE version

The steps of this video shows you how to:

Download the tools
Install the tools
Start up eclipse

I’ve put the presentation on YouTube, but because of Adobe Captivates lack of support for the YouTube formats (they don’t support HD videos export), the quality is rather poor. You can (if you tolerate the back button on you browser) also view a high-quality version by clicking on this link (HIGH RESOLUTION DEMO).

Configure Tomcat

In the previous video we came as far as starting up Eclipse. We have Tomcat and CXF installed, and we now need to configure Eclipse so that it knows where our Tomcat and CXF installations are. Please note that if you change workspace, you will have to do this all over again (or save the preferences to a file, then restore them in your new workspace).

Again, I decided to do this through a video demo. If you want a high quality version click on this link (HIGH RESOLUTION DEMO).

Test Tomcat

The next thing we'll do is to ensure that Eclipse and Tomcat is properly installed and configured. I'll just create a simple servlet to do the test. You may want to skip this demo all together if you want to get to Web Services as quickly as possible.

Again I've created a video to demo the concepts. Again, the YouTube version is of inferior quality, but if you want to see it in high def... here's the link (HIGH RESOLUTION DEMO).

Configure CXF

Tomcat is setup to run Axix web service framework by default. We want to change it to use the CXF framework that we downloaded. In this demo I'll show you have to do that. The steps involved are:

Add CXF to the Web Service Runtimes
Configure Tomcat to use CXF

As before you can watch this at YouTube (picture below) or in high definition here (HIGH RESOLUTION DEMO).

Creating a Web Service

All the steps up to here had nothing to do with Web Services. The steps prior to this has been about setup (download the tools, configure Eclipse, testing to ensure everything works).

We’ll create the simplest of Web Service… We’ll call it DeepThought. The service will have one method

public String whatIsTheAnswer(String nameOfInterviewer);

The web service implementation will provide the answer to the universe, life and everything in a String.

If you want a high definition version of this demo, here’s the link (HIGH RESOLUTION DEMO).

Conclusion

This blog entry showed you how to download, configure and develop a web service using Eclipse, Tomcat and Apache CXF. The process involves quite a few steps. You have to:

Download the tools (Apache Tomcat, Apache CXF, Eclipse)
Install the tools (Mostly a matter of unzipping the archives you downloaded)
Configure Eclipse (Where is Tomcat and CXF, Configure Tomcat to use CXF, etc)
Create a Java implementation of your service
Build the web service

Although I’m sure in a few months the description will be slightly outdated (new wizards in Eclipse, etc.), I hope you’ll find the tutorial useful.

Search Engine and Indexing of Dynamic Content

2010-10-07T08:45:00.001-07:00

What’s the problem?

When loading a dynamic site using GWT, Dojo, YUI or any other Ajax based approach, it is difficult (and sometimes impossible) for the search engine to obtain your content for indexing. The initial load of the site may only contain a simple HTML element (often a div or span tag). The tag will then be filled in by JavaScript using asynchronous calls to the server (Ajax) after the page has been loaded.

The search engines never got to see the real content.

Example

I’ll use our course catalog as an example. You can browse the course catalog by going to http://www.scispike.com/training-general/course-list.html.

We have 100’s of courses and the list of courses changes frequently. It is essential for us that the site allows us to modify course outlines and course categories with minimal hassle. In practical terms this means the course descriptions and their outlines must be retrieved from a dynamic source. The source for the page above has two dynamic areas. The course catalog (where the list of courses and their categories are displayed) and the search box for courses.

The course list is rendered from a persistent data source that is easily modifiable. That is, we have the ability to upload courses without redeploying the site. To make this happen, we have written some client side JavaScript (using GWT, we actually wrote this in Java) that when the pages has loaded will dynamically fetch the current course catalog from the persistent source.

The problem, of course is… what does the search engine see?

...
<div id="courseCatalog"></div>
...

This is literally all they see! This is pretty tragic for us because it means that web crawlers can’t find our web pages and hence nobody will find our courses.

Sitemaps

The web crawlers allow you to setup an XML file called a sitemap. A sitemap is a simple XML format that lists all the pages on your site and gives other search hints to the search engines. It is a really simple format (you can read more about it at http://www.sitemaps.org/). The sitemap files looks like this:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>http://www.scispike.com/about/about.html</loc>
<lastmod>2009-02-22</lastmod>
<changefreq>monthly</changefreq>
<priority>0.7</priority>
</url>
<url>
<loc>http://www.scispike.com/about/careers.html</loc>
<lastmod>2010-10-06</lastmod>
<changefreq>weekly</changefreq>
<priority>0.1</priority>
</url>
...
</urlmap>

The file contains a description of all the files you want the search engines to index. It also contains hints as to how frequent the pages change (tells the search engine how often they ought to crawl the site) and the relative priority of the pages (a hint to say some pages are more important than others).

Each of our course outlines can be obtained by a unique URL. This allows us to define the sitemap containing all the pages on the site.

To make this work, we have to make sure we also generate the sitemap from the persistent source. I hence created a special servlet that reads the content from the database and generates the sitemap on the fly.

All we had to do now was to go to the main search engines and submit the URL of the sitemap. I did it yesterday and I noticed that some time before the search engines start the crawl, hence I’m waiting eagerly on the result, but at least in theory this should work well.

Convenience list

When I went to Yahoo to submit my sitemap, I could only find an interface to submit individual URL’s. It would be tragic if I had to go to the Yahoo site to submit a new URL every time we added or changed a course, so I decided that we really need to provide a way for crawlers that do not use sitemaps to find our outlines.

I want humans to use our course-list. It has all kinds of fancy features (try out the search box and look at the auto completion), however, I want the crawlers to see the pages using standard HTML. What I decided to do is to also generate a simple list. The list is generated on the server. The list has no frills (it is as simple of a page as possible), but it is navigable for crawlers from the index page.

You can see this list if you know the URL (check out this URL), however, you would have to know the URL to find it. Our front page has a hidden link defined as follows:

<div style="display: none">
<p><a href="all-links.html">All pages on this site</a></p>
</div>

A crawler will find the link and hence chase down all the outlines. This allows me to submit only one page (http://www.scispike.com). When pages change, the all-links.html also change. I guess I could have avoided the hidden link on the front page and rather submitted the hidden page, but I think this is better as I am pretty crawlers will (eventually) start finding the front page and new search engines should hence also pick up the hidden URL.

Summary

When creating a dynamic site, you have to pay particular attention to how your dynamic content is published to the search engines. A typical Ajax application will not be found by the search engines because the content is not provided in the initial html download rather rendered after the page has been loaded or after some interaction on your site.

You may submit a sitemap to the search engines that informs the engine of content not reachable through site navigation by crawlers (you would typically also provide the content reachable too).

In the example above, I was in luck, because all the content is available through a unique URL. Only the catalog is not crawlable. To solve this unique situation, I used an approach as shown below:

I created some server side code that reads the dynamic content and generates a sitemap on the fly. The server also generates a static page (actually it is a dynamic page, but it will appear to the crawlers as static) containing all the links.

I did not want the list of links to be visible to the users (they have a nice dynamic course catalog) that they can use, so I provided a hidden link to the file that the crawlers only would use.

GWT, Google App Engine and SciSpike (Part 0 of ??)

2010-10-06T16:16:00.001-07:00

This week (and most of last week) I’ve been developing a GWT application for our new venture SciSpike (www.scispike.com).

The team at SciSpike are all very experienced architects/developers, so the selection of technology to use was quite difficult. Pretty much every options were available to us. We all are used to architecting and teaching classes on various web technology. We discussed using:

All technologies that we master and all good choices for building a dynamic site. Eventually we settled for GWT and Google Application Engine (GAE).

Looking back, I think we made a wise choice for a couple of reasons:

We managed to go from conception to a complete site in less than one week (speaks volumes of the productivity in GWT)
We deployed the site with no hassle (using GAE and deploying the site to the Google cloud)

We did run across a set of issues that I though others may benefit from. It is my goal to share these issues with you in a few of the next blog entries. In particular, I’ll discuss.

Search Engine Optimization with dynamic content
Combining GWT with generative approaches
Using the Google Site Search

Visionaries in Software

2010-03-09T17:26:00.001-08:00

This week I’m documenting architecture for a system where my company has been involved in the design and development of the core framework. I was not originally involved in this project, but have later contributed to the framework.

It always amazes me to see the pockets of cleverness found at small companies and government organizations. I do expect to see clever solutions when working with the well established software organizations. After all, they hire from top shelf and are behind most of the groundbreaking research in the software field. However, in recalling the last 20 years of consulting, the most memorable architectural solutions all come from small companies or from pockets within the government.

The solution I’m documenting now is a deployed application at the State of Montana. It is using software techniques that many software organizations are still experimenting with. The system has code generators based on state-of-the-art domain specific languages, they are using a sophisticated set of data cashing using a variation of SDO (Service Data Objects), the build process is automated with Maven and Hudson, and so on…. Many solutions that would make for a nice study and learning experience for most software organizations.

So what is it that makes some organizations excel in software? Thinking back, I can find only one common factor! At the top there is a visionary architect or manager. Usually, it requires only one that has the power and is willing to push the boundaries.

The project I’m working on now is not an exception. When I was first brought in to look at their architecture, I noticed that much of the manual work could be automated with a code generator. A lot of the code was already written, so I was hesitating a bit to suggest building a DSL and a tool. However, it seemed the best option, so I felt I had to propose it (If it is one thing I’ve learned for through nearly 20 years of consulting it is “there is NEVER a good reason not to suggest what you think is the best solution, NEVER!”).

I was expecting to have to spend hours/days to explain and also considered my chances of success miniscule. However, never underestimates the visionaries…. Without much discussions at all the advantages were seen immediately and the decision to move forward taken within minutes.

To all the visionaries out there, I applaud you!

Deploying the code generator with Xtext/Xpand

2009-12-14T12:18:00.001-08:00

This week I deployed a DSL with its corresponding code generator for a client. The DSL and corresponding editor was written in Xtext and the code generator in Xpand/Xtend.
I was thinking through various options for deploying the editor and code-generator and in the process I investigated the options in Xtext. At first I was very disappointed in the options, but then Peter Friese gave me a short lesson (for which I owe him a(nother) beer :)).
Xtext has a default suggestion for how to integrate the editor and the code generator, but I’m not sure if the option is easily obtainable through the documentation. To help others, I thought it would be a good idea to write down what I learned in this blog-entry.

Description of Problem

After you have created your DSL, its editor and the code expansion templates, you need to decide how to deploy your solution to the users of the DSL. The deployment mechanism is the use of Eclipse plug-ins, but there are a few variables. In particular, it is not clear how you connect the editor and the code generator.
There are a few common scenarios for how to perform the code generation. They are not mutual exclusive (that is, you may want to support multiple options):

Activation of the code generation from the editor, e.g.:
- The user right-click inside the editor and selects Generate Code
- The user right-click on the DSL file and select Generate Code
- A wizard
You provide a Eclipse Modeling Workflow that then selects the model file and calls the expansion templates
- Ideally we should now have a way for the DSL user to create this file (a new wizard)
Incremental build
- Hook the code generation up with the incremental build in Eclipse as for instance the JDT does (When a java file is saved, the incremental build automatically invokes the compiler to produce the class files)
Headless invocation of the code generator targeted for build tools such as:
- Maven
- ANT

The question is, which one do you choose and what do you have to do to achieve it?

What Xtext Supports!

Xtext provides one integration option for the code generation in its current version. This option involves hardly any work at all. When you use the Xtext wizard to create a new Xtext project, the mwe (Eclipse Modeling Workflow) that Xtext create for you has an option for generating a '”new wizard” that is commented out (why it is commented out, I don’t know).

<!-- project wizard (optional) -->
<fragment class="org.eclipse.xtext.ui.generator.projectWizard.SimpleProjectWizardFragment" generatorProjectName="${projectName}.generator" modelFileExtension="sdol"/>

By enabling the new project wizard, the Xtext generator will build an extension in the U/I that constructs a “new project” wizard. That is, when the U/I plug-in is installed, you’ll be able to create a new project particularly tailored to generate code. You can get to this “New Project” wizard by selecting:

File => New… => Project…

and then navigate to the Xtext tab. The option will be called “<MYDSL> Project” where MYDSL is the name of your DSL. In the wizard below, you’ll see “SDOL project”, which happens to be the name of a language that I recently created.

After having chosen to create your DSL project type, you’ll be presented with the generated “New Project” dialog that basically just asks you to give the project a name:

The resulting project has been configured to enable code generation. It is a plug-in project and it uses the plug-in dependencies to load the required plug-ins required to execute the Xpand/Xtend templates you created for your DSL using Eclipse Modeling Workflow. The plug-ins required are:

com.ibm.icu
org.eclipse.xtext.log4
org.eclipse.xtext
org.eclipse.xtext.generator
org.eclipse.xtend
org.eclipse.xtend.typesystem.emf
org.eclipse.xpand
org.apache.commons.logging
de.itemis.xtext.antlr
org.eclipse.emf.codegen.ecore
org.eclipse.xtend.util.stdlib,
<MYDSL>.generator

You can configure the way the way the project is initialized using some expansion templates in the U/I plug-in. In the U/I project you’ll find an Xpand template called <MYDSL>NewProject.xpt. This file determines the content of a new project. The default expansion template is shown below, but you can obviously change this template to be whatever you want it to be.

«IMPORT my::project::dsl::ui::wizard»
«DEFINE main FOR SDOLProjectInfo»
«EXPAND model FOR this»
«EXPAND workflow FOR this»
«ENDDEFINE»
«DEFINE model FOR SDOLProjectInfo»
«FILE "src/Model.sdo"-»
/*

* This is an example model
*/
type String
entity Leaf extends Composable {
property name: String
}
entity Composite extends Composable {
property content: Composable[]
}
entity Composable {
}
«ENDFILE»
«ENDDEFINE»
«DEFINE workflow FOR SDOLProjectInfo»
«FILE "src/Generator.mwe"-»
<workflow>
<property name="modelFile" value="Model.sdo"/>
<property name="targetDir" value="../src-gen"/>
<workflow file="workflow/SDOLGenerator.mwe"/>
</workflow>
«ENDFILE»
«ENDDEFINE»

When the Xtext Approach does not work…

The approach above should work well in most cases. There is one case where the approach above is problematic (which of course happens to be the one I’m currently are working one). The approach above only works if you are targeting the same Eclipse version as the one you’re using as your IDE.
In my case, my client is targeting an older version of eclipse than the one they use as their IDE. This is not an uncommon scenario. When building Rich Client Platforms (RCP) in Eclipse, you typically want to setup the target platform to a '”fixed version”, but at the same time, you need to leverage the latest Eclipse tools. Because the approach mentioned above leverage the Eclipse dependency mechanism, Eclipse will look for the required plug-ins in the target platform (as opposed to the current IDE).
I actually think the current Xtext approach is a bit of kludge because it uses a run-time dependency setup for what should be resolved at compile/build time , but I can see the convenience of the approach. Also, it probably works for most projects.

How to Work around the Kludge?

To work around the dependency problem I have at my client site, I’ll have to ensure that all the code execute using the IDE plug-ins. That means, I cannot setup the code-generation project using the eclipse plug-in dependencies. I have two choices:

Manually managed dependencies
- Setup the code generation project as a simple Java project
- Setup the classpath to point to each of the required plug-in jar files mentioned above
Execute the workflow from within the installed IDE plug-ins
- Write one (or more) U/I extensions to <MYDSL>.ui plugin that activates a code generation task from within the installed plug-in
- Write preference pages for configurable options in the plug-ins for options that are typically setup through the workflow file (e.g., target directory for the files, model file)

I’m probably going down the option 2 path and I’ll write up something on my blog when I’m done. Option 1 is of course the easier of the options, but it requires a rather fragile setup of the classpath.

Other Common Code Generation Options

Here are some other code-generation options that you may want to consider:

Incremental build of Eclipse
Maven
ANT
Integrated customized

Incremental Build

It will take too long to write up this option. It requires understanding of how to setup builders, natures and also how to read resource deltas in Eclipse. The good news is that the Xtext team is currently working on this option. You can read about the plans for this here.
In many projects, this will probably be the preferred option as it fits well with the way Eclipse works for most other tools/languages. That is, as soon as you save your DSL file, eclipse would automatically launch your code-generation script to generate code in the background.

Integration with Maven

Many rely on Maven to do their build. I’ll not argue for nor against this approach, but it may very well be that you need to integrate a code generation task with Maven. Karsten Thoms has written up an article of how to integrate Xtext with Maven that you can read here.

Integration with ANT

Another popular language for build scripts is ANT. You can read about the ANT integration here.

Conclusion

When using Xtext, you need to devise a plan for how to integrate code-generation for the users of your DSL. That is, the user must be able to initialize the code-generation step after having produced the DSL file using your editor.
There are several options for generation. You may want to let the users activate the code-generation in their IDE. You may also want to provide a method for them to run their code generation as part of a build script. All options are achievable in Xtext.
Xtext supports a default integration of code-generation using the “New Project” wizard to create a project setup for code generation. This project uses the plug-in dependencies to activate required plug-ins. This will not work if you target a different eclipse run-time version than the one used for your IDE.
The Xtext team is planning an integration using the incremental build which may work around the Eclipse versioning problem.
You can integrate code-generation step into the editor (e.g., context menus etc.). Integration is relatively straight forward if you know how to write U/I plug-ins as the workflow can be executed from memory (no mwe files).

A course on Writing Extensions for Plug-ins

2009-11-23T19:04:00.001-08:00

I created a e-learning module for how to create plug-in extensions for Eclipse. It is a sub-module from our course how to write eclipse plug-ins that I’ve not yet integrated. Many students have asked for this, so I though I’d make it available for them here.

Anyway, if you are interested, click on the image below and it will bring you to the site. I did notice that my Internet Explorer don’t like the link. I’m not sure why, but then I tried it on another computer with IE, and it worked fine… If you have problems viewing the video, let me know.

Please let me know if you have suggestions for improvements.

Xtext vs. EMFText: Code generation

2009-11-23T17:11:00.001-08:00

Introduction

This post is going to be very short as there comparison here is rather trivial. Both tools can be integrated with any of the transformation frameworks available in Eclipse.

EMFText does not include transformation nor suggest any specific transformer.

Xtext integrates tightly with one of the transformation frameworks, but can be reconfigured to use others.

Xtext and Xpand/Xtend

Xtext provides an integrated model-to-text (M2T) code generation framework called Xpand. Xpand is a separate Eclipse project (part of the M2T in Eclipse). The integration is relatively seamless and the wizards of Xtext create a separate project dedicated to code generation.

Xtext also provides an integration with Xtend. Xtend can be used to extend Xpand and it can also be used as a model-to-model (M2M) transformation framework.

Xpand and Xtend are mature transformation tools that you can read about here. I’ll not discuss the Xpand nor Xtend framework further in this blog (I’m thinking of writing up a comparison blog of the various M2T framework later).

If for some reason you’d like to use another transformation framework, you can do so with Xtext, although you have to spend some time deconfiguring and reconfiguring your project.

EMFText and Transformers

The short story is that EMFText does not include a transformation framework , but I speculate that this is deliberate!
There are several options for transformation from the abstract syntax (that is the ecore model) in Eclipse (and also outside eclipse) and EMFText allows you to choose whichever toolset you’d like.

For M2T, the obvious choices are Xpand, JET, or Acceleo.

For M2M, there are also many choices (including Xtend). For more information of the M2M projects for Eclipse, take a look at the M2M portal at Eclipse.

Which Approach is better?

If you don’t have a preference for transformation tools, you may benefit from the tight integration that Xtext provides with Xpand and Xtend. The new project wizard creates a complete setup using Xpand and Xtext.

However, if you do have a preference and this preference is not Xpand and Xtext, you may find the EMFText a better starting point as you don’t have to 'deconfigure’ your project to remove the Xpand/Xtend dependencies.

As I said when comparing the development processes, it is unfair to say that one approach is better than the other. They’re just different.

On a personal note, some of my code generators are too complex to use M2T or a trival M2M transformation. I have to run the model through a chain of M2M transformation before the final M2T, in these projects. For these projects I could use Xtext and reconfigure the projects or just simply use EMFText.

Conclusion

Again we’re seeing a slightly different philosophy when looking at EMFText and Xtext. EMFText does not provide any particular code generation framework (or other transformer technologies) and you decide which framework to use.

With Xtext you’ll pay a penalty if you want to use a different framework than Xpand/Xtend, but if you use Xpand/Xtext, configuration is automatic.

In EMFText you would have to configure the code generation framework yourself, but you never have to ‘deconfigure’ the project :)

Xtext vs. EMFText: Development Process

2009-11-23T13:10:00.001-08:00

Introduction

In my previous post, I compared Xtext and EMFText. I pointed out a few differences that may help you select one tool over the other. In the next posts I'll go into each of the points in more details. In this first in-depth comparison, I'll be focusing on the differences in the development processes between the two tools.

Two kinds of DSL's

If I take all the DSL's I've defined, I can partition them into two groups based on what was the main focus when discussing the language:

Abstract Syntax.
In some projects, I developed multiple concrete syntaxes. When this is the case, the language is most often designed using the model for the abstract syntax. This leads to a process where we first design the language features using the abstract syntax, then later we'll discuss how we want the changes to the abstract syntax to be reflected in the concrete syntax.
In short: Abstract syntax first à Concrete syntax derived
Concrete Syntax.
In other project I had no intention of building more than a single concrete syntax. In these projects, the abstract syntax is usually developed after we've made decisions on how to change the concrete syntax.
In short: Concrete syntax first à Abstract syntax derived

When comparing the Xtext and EMFText, one major difference between the tools is that the natural development process for Xtext is to first develop the concrete syntax and let the abstract syntax be derived. When using EMFText, the natural development process is to first develop the abstract syntax, then define the concrete syntax.

Development process for EMFText

The figure above shows a high level outline of the EMFText development process. The matrix shows two dimensions. The X-axis shows if the technology used is original or if it the tool is derived from some other technology. The Y-axis shows the process for abstract vs. concrete syntax.

The process can be described in text as follows:

Develop your ecore model. For this you can use any tool, none is prescribed, none is supplied. There are many tools available for building EMF models. For more details of how to create an EMF model, check out the EMF site or the following presentation.
Develop the concrete textual syntax using the tools provided by EMFText. The tool provides a language for defining the concrete syntax of a already supplied ecore model (containing the abstract syntax)
Run the EMFText provided generator. This generator relies on ANTLR to build a parser.

Development Process for Xtext

The figure above shows the main processes of Xtext. Notice that process is quite different from that of EMFText. Although Xtext depend on EMF and uses ecore models for abstract syntax, you would typically let Xtext generate the abstract syntax from the Xtext language.

There is also a way to use existing abstract syntaxes with Xtext (although, that requires a bit of configuration). You can read about how here.

Which approach is better?

I'm sure one may construct an argument arguing for or against either approaches. My personal view is that neither approach is superior to the other.

My recommendation would be to evaluate how well the tool fits your experience/goals.

Factors that may influence your decision

How many concrete syntaxes do you need?

If you develop multiple concrete syntaxes, you would typically want to focus your design around the abstract syntax. One may argue that one always should, but it is my experience that unless you have two or more concrete syntaxes, the discussion will be around the concrete syntax.

Both tools supports focusing on abstract syntax first, but this approach would fit more natural with the approach of EMFText.

If you intend to produce one concrete syntax only, you may find the approach of Xtext better. You only need to maintain one file that contains both abstract and concrete syntax definitions.

Do you already know EMF?

You cannot use EMFText unless you also learn EMF. You need to choose an approach that first produces ecore models. If you already know how to create ecore models, the EMFText may be more appealing than Xtext.

Conclusion

Both Xtext and EMFText are tools that help you create a textual syntax for a DSL. One of the main differences between the two tools is the proposed development process. Xtext provides one language from which it derives both the concrete and the abstract syntax. EMFText relies on EMF based tools to create an ecore model that they then 'annotate' with definition of a concrete textual syntax.

If your language requires multiple concrete syntaxes and you already are familiar with EMF, then the EMFText may be the best fit. If you are defining the single concrete syntax, Xtext may be a better choice because you can derive the abstract syntax directly from Xtext and also no knowledge of EMF is required.

The selection criteria for the two tools are rather fuzzy. You may separate the abstract syntax from the concrete syntax in Xtext also; hence, you can easily support multiple syntaxes in Xtext. Also, when generating code from Xtext, you typically will have to understand EMF and ecore, so the advantage of hiding ecore may be mute.

I would recommend that you understand the differences, evaluate your own (or your organization's) background and make a decision. Whether you select Xtext or EMFText, you gained a fantastic tool.

Xtext or EMFText… that is the question

2009-11-20T08:00:00.001-08:00

I just finished a project building a textual DSL and a code generator for a client and after some intensive investigation, Xtext seemed the best alternative.

Today, I learned that EMFText just released a new version (1.2), and it looks very promising. It seems I'll have to investigate the two tools and make a comparison now. I'm sure both tools are good and that my choice of Xtext will stand, but building languages have become an important part of my business, so I better get on top of all technologies.

At first look, the two approaches look very similar.

Similarities

Both tools seem to provide the exact same result. You can define a textual language; you'll end up with a good eclipse-based editor for your language that provides the essential features we have come to expect for such an editor. That is:

Syntax highlighting
On the fly parsing with error/warnings
Auto-completion
Outlining

Both tools use EMF for their abstract syntax. Both tools seem to rely on code-generated ANTLR for the parsing of the concrete syntax. Both tools provide excellent editors for editing syntax files. Both tools seem reasonably well documented.

What then is different?

There are a few things that popped up on first review. I thought I write them up immediately… Maybe the Xtext and EMFText teams can keep me straight as I make the comparison?

Different process

In Xtext you would typically define your abstract and concrete syntax together. Xtext does support an externally defined abstract syntax, but that is not its default process.

EMFText assumes that you already have defined your abstract syntax in EMF. Using the EMF-based abstract syntax, you create the concrete syntax by creating a textual definition in a language defined by EMFText. The concrete syntax is defined in a file and contains a reference to the abstract syntax definition, a section of options (mostly for code-generation, similar to what you normally see in the gen-model of EMF if that helps you), and then the EBNF grammar. Here is the suggested development process for EMFText:

http://www.emftext.org/images/8/82/Emftext_process.png

I don't think one approach is superior to the other. There are times when the abstract syntax is dominant where the EMFText approach seems cleaner. Other times nobody cares about the abstract syntax and the Xtext approach seems more appropriate.

Code generation options

EMFText does not tie in to a code generation framework. This is not an issue for experienced Eclipse programmers familiar with the modeling tools. One may for instance easily integrate it with JET or any of the other M2T or M2M projects (after all, the model is parsed and available in it EMF).

Xtext on the other hand by default integrates with Xpand and Xtend. This means that your project is immediately ready for a code generator.

If your goal is to generate code and you do not have a lot of experience integrating modeling technologies in Eclipse, Xtext will probably allow for an easier learning curve (e.g., a single documentation spanning language definition and code generation). However, there are projects where the only goals I've had were to generate the EMF content.

Again, I would not claim that one approach is superior to the other… They are just different.

Predefined languages

EMFText comes with a few predefined languages that you can download from the Concrete Syntax Zoo. This may potentially be an advantage if you want to extend existing languages or simply want some examples for inspiration.

Documentation and tutorials

There is a lot more documentation for Xtext than for EMFText right now. That doesn't necessarily mean that Xtext is better documented. I'll have to go through an exercise of using EMFText on a real project before I judge, but my first impression is that Xtext has better documentation.

More later

That's all for now. When I've collected all my data and have some time to write it all up, I'll make a comparison matrix :)

Xtext Valueconverter

2009-11-15T19:57:00.001-08:00

I thought I'd share another little trick in Xtext. This trick is actually documented well at various places on the web, but to be able to do what I wanted to do, I had to pull information from many different locations. So… I thought it may be a good idea to collect the tips here and share them with my readers.

What I want to achieve?

Occasionally, I run into parsing problems that cannot be solved directly in Xtext. The example I'll show you is to parse a cardinality expression.

What I want to parse is a UML-ish cardinality in the form:

[lowerBound…upperBound]

E.g.:

[0..1], [1..10], [1..*]

The cardinality expression has one tricky part, how do we handle the '*'?

In my case I have a model (or meta-model if you so prefer) where I want to create a cardinality object that has two attributes, both being of type int. The lower bound is typically 0 or 1 (theoretically, any number, but practically either 0 or 1). The upper bound can be 1, n (>1) or '*' which I translate to -1 in my model (similar to what ecore does if you know EMF).

The first (erroneous) attempt

It is tempting at first to try to do something like this:

Cardinality:

'[' lowerBound=INT '..' (upperBound= INT | upperBound=*)']';

But that will not work. The problem is of course that there is no way to assign a value for upperBound to -1. There has been some talk (in various newsgroup entries) to extend Xtext this way, but as it is now, we need to use another trick.

Value converters

The trick to make this work is to use a value converter. Let me jump straight to it and show you the Xtext fragment.

Cardinality:

'[' lowerBound=ElementBound '..' upperBound=ElementBound ']';

ElementBound returns ecore::EIntegerObject: // datatype rule handled by a value converter

'*' | INT;

Notice in the example above that we have specified that ElementBound returns an integer object (ecore::EIntegerObject). The framework already knows how to convert INT to and integer, but it would fail if we entered '*'.

We then have to provide a value converter that can transform the parsed text (an INT or '*') to an integer. We do so by writing a simple Java class.

Writing and attaching the value converter

To create the value converter you create a new Java class. This class can be named anything you want. In Xtext you just have to make sure it is available in the classpath used when parsing. In practical terms, that means you'll place it somewhere in the source directory of your parser project. The example below, I've placed the value converter in a package called 'com.inferdata.converter'. I've named the class 'CardinalityConverter', but as I said, the class name and package name is your choice.

The class needs to implement the following one method. The method constructs a value converter for the particular rule that needs custom conversion.

In general, you would also want to ensure that you subclass org.eclipse.xtext.common.services.DefaultTerminalConverters. The reason is that this class already knows how to transform cross-links, INT, String, etc that you probably rely on elsewhere.

Here is a working converter:

package com.inferdata.converter;

import org.eclipse.xtext.common.services.DefaultTerminalConverters;
import org.eclipse.xtext.conversion.IValueConverter;
import org.eclipse.xtext.conversion.ValueConverter;
import org.eclipse.xtext.conversion.ValueConverterException;
import org.eclipse.xtext.parsetree.AbstractNode;
import org.eclipse.xtext.util.Strings;

public class MyLangValueConverter extends DefaultTerminalConverters {
    @ValueConverter(rule = "ElementBound")
    public IValueConverter<Integer> ElementBound() {
        return new IValueConverter<Integer>() {
            public Integer toValue(String string, AbstractNode node) {
                if (Strings.isEmpty(string))
                    throw new ValueConverterException("Couldn't convert empty string to int", node, null);
                else if ("*".equals(string.trim()))
                    return -1;
                try {
                    return Integer.parseInt(string);
                } catch (NumberFormatException e) {
                    throw new ValueConverterException("Couldn't convert '"+string+"' to int", node, e);
                }
            }

            public String toString(Integer value) {
                return ((value == -1) ? "*" : Integer.toString(value));
            }
        };

Explaining the converter code

The first method is a factory method to create an object that converts the result of the ElementBounds rule to an Integer.

@ValueConverter(rule = "ElementBound")

public IValueConverter<Integer> ElementBound() { … }

There are several rules here that we have to follow:

The method must have an annotation of type ValueConverter. This rule must specify one attribute, the rule to which the method applies. In our case we are providing a converter for the rule 'ElementBound'
The method must return an instance implementing the IValueConverter of the type we want to construct. In our case, we are converting to an Integer, hence, we must return an instance that implements IValueConverter<Integer>
The method must be named the same as the rule. In our case, that means that the method must be named 'ElementBound'

The IValueConverter<Integer> interface dictates that we have to provide two methods:

Integer toValue(String s, AbstractNode n)
One to translate from a string to an Integer (Integer, because that was our generic type parameter). We also get access to the AbstractNode in the parse tree (which we will not need in this example)
String toString(Integer value)
The inverse operation of 'toVlaue'. Translate from an Integer to a String

The internal algorithms in the methods are rather straight forward, and I will not explain them here.

Configuring the value converter

Xtext uses GUICE to configure the value converter. You can read about the Google-Guice at Google's site. Basically, GUICE is used to inject the value converter into the framework.

The file we need to modify to inject our new value converter is (typically) placed in the same directory where you have your xtext grammar file. It is called <NAME_OF_YOUR_LANGUAGE>RuntimeModule. It is important that you place it here as the other places that use GUICE are in generated code and hence will be overridden on subsequent code generation cycles.

package com.inferdata;
import com.inferdata.converter.MyLangValueConverter;
import org.eclipse.xtext.conversion.IValueConverterService;

/* Use this class to register components to be used within the IDE.
*/
public class MyLangRuntimeModule extends gov.mt.dli.AbstractSDOLRuntimeModule {
    @Override
    public Class<? extends IValueConverterService> bindIValueConverterService() {
        return MyLangValueConverter.class;
    }
}

As mentioned, Xtext here relies on GUICE to inject the code. What we've done is that we've replaced the default value converter with our own value converter called MyLangValueConverter. This is done with the bindIValueConverterService.

Conclusion

In some cases, it is necessary to build custom value converters for some of the rules in Xtext. This need may go away (or lessen) with new versions of Xtexts.

To attach a new value converter you have to:

Modify the xtext grammar
- Define the rule specifying a 'returns' clause
Create a value converter factory class that instantiates a specialized value converter for each of the rules that needs customized conversions
- Annotated factory method with the annotation @ValueConverter(rule = "<THE NAME OF YOUR RULE>")
- Name the method after your rule
- Return an instance that implements IValueConverter<TYPE> where, TYPE is whatever your return from the xtext specialized clause
- You typically want this class to extend the default factory class provided by the Xtext framework
Provide a specialized implementation of the IValueConverter (we used an anonymous inner class in our example)
- Implements toValue that converts from a string to the type you require
- Implements toString that converts from the value back into a string
Reconfigure the parser using Google's GUICE
- Provide a bind statement that binds the value conversion service to your specialized factory class

How to Write an Extension in Xtend for Xpand in Xtext

2009-11-15T09:55:00.000-08:00

I ran into a problem in Xtext some weeks ago. I wanted to generate a serialization id for my generated Java classes. After some dumbfounded attempt to do so in Xpand, I realized that Xpand does not allow for an arbitrary Java algorithm to be interspersed in the (almost) declarative Xpand template language.

I asked the generous Xtext team for help, and it became clear that I would have to create an extension using Xtend. I’ve never done that before, but the Xtext team pointed me to the documentation. After having read the documentation forwards and backwards, I was still confused. I think the presumed knowledge did not quite fit my profile. To help the Xtext team and maybe others with the same background as I, I thought I'd share my experiences.

What I wanted to achieve

I want to use Xpand to generate some Java code that use Java serialization. One of the foundations for Java serialization is that you provide ‘long’-value. It doesn’t much matter what the number is, but it better change if you change the class. This is to ensure that previously serialized data no longer is compatible with the new Java code. Hence, when writing my code expansion template, I would have to generate a sensible number that would be different given two different input models.

The template looked something like this:

...
public class «modelObject.name» {
private static final long serialVersionUID = «HOW DO I DO THIS???» ;
...

In Xpand you there is no way to inject arbitrary Java code (as say in JET or Velocity templates), however, you can achieve this in a clean way using Xtend.
Reading the Xtend documentation (see link to above), it was not clear to me exactly how to achieve this. In particular, I was not sure where to put the files and how to link everything.

What was clear was that it would be possible for me to extend Xpand such that I can now use a simple new function. That is, after I’m done, I should be able to write my template as this:

…
public class «modelObject.name» {
private static final long serialVersionUID = «generateSerialUUID(modelObject)»;
…

How to extend Xpand

Here is a list of facts that you may want to keep in mind as you read through the rest of this blog entry:

Xtend allows you to extend Xpand in two different languages. You can use OCL (or a derivative of it). This is very simple and the preferred way. The other is to declare the extension in Xtend and implement it in Java. This is a bit trickier, but not too difficult after you’ve read this (I hope).
The Xtend declaration or definitions are defined in files with extension ‘ext’. If you are using Xtext, you would want to place them in the generator project. You may put them in any subdirectory of the main source (default called ‘src’).
If you implement part of your extensions in Java, you should also place the Java code in your generator project.

Step 1, define your Xtend file

The first step is to create a file that defines your extensions. You can place the Xtend file anywhere in the classpath defined when you run Xpand. The natural place to place the file(s) if you use Xtext would be in your ‘generator’ project src directory. In my case I placed them in ‘src/extensions’. The file must have the extension ‘ext’. In my case, I created a file called JavaSerialUID.ext.

Step 2, declare the content in the Xtend file

As mentioned earlier, there are two languages that you can use. This fact is clear in the official documentation, so I’ll not dwell with this much. I’ll just focus on creating a Java extension. My extension generates UID’s as mentioned earlier, and the content looks like this:

import sdol; // import my dsl
/**
* Java extensions mapping an Xpand function to the Java code
*/
String generateSerialID(sdol::Type t) : JAVA
com.inferdata.sdol.extensions.JavaUtil.javaSerial(com.inferdata.sdol.Type);

Notice that the content of the Xtend file for Java type extensions is really nothing but a simple mapping. It is basically stating that we want to extend the Xpand language with a new function called ‘generateSerialID’ that requires an sdol::Type (sdol is the name of the language I’ve created, and Type is an object type in that language). It is also mapping this new function in Xpand to an implemented function in Java. This code is placed in the package ‘com.inferdata.sdol.extensions’. The class name is ‘JavaUtil’. The method takes one argument, that of type ‘com.inferdata.sdol.Type’, that is we’re providing the fully qualified name of the type (e.g., if you wanted to pass a string, we would have to specify ‘java.lang.String’). Also note that we promise that the result of the invocation will be a string object.

An important little digression may be in order here. Notice that the declaration refers to my model object ‘Type’ in two different ways. It is using the Xtend language to refer to the Type in the declaration. That is, in Xtend my type is called ‘sdol::Type’. In Java, the same model object is being referred to as com.inferdata.sdol.Type. The Xtend reference is identical to the one we would use in Xpand, the Java reference is based on the generated EMF code.

Step 3. Create the Java implementation

Now we need to write the Java code we want executed when someone use our function.
My Java implementation for generating a serialization ID is not important and whatever algorithm you may need would obviously be different, but here is a simplified version of what I did (generated a hash code by xor’ing the names of all features):

package com.inferdata.sdol.extensions;
import com.inferdata.sdol.*;
public class JavaUtil {
public static String javaSerial(Type t) {
long hash = 0L;
for ( Feature f : t.getFeatures() )
hash ^= f.getName().hashCode();
return Long.toString(hash);
}
}

The important thing to note here is that the class and method name maps EXACTLY to that declared in the Xtend file.

Using Xpand after the extension

After you have finished your extension, you can now use the new function from within Xpand. The Xpand editor will even help with auto-completion.
The first thing you need to do in your Xpand file is to import the newly created extension. You do this by declaring an «EXTENSION» element.

«EXTENSION extensions::JavaSerialUID»

This declaration above has to be exact also. The ‘extensions’ is the directory where I placed my extension file. ‘JavaSerialUID’ is the name of the extension file (the file name is really ‘JavaSerialUID.ext’, but we drop the file extension).
Next we need to know one more thing… The extensions can be invoked two different ways. You may invoke the function in a pseudo object oriented fashion. That is we’ll write __OBJECT_REFERENCE__.expansionFunction(…), or we may use a more functional syntax, expansionFunction(__OBJECT_REFERENCE__, …). The variation is nothing but syntactic sugar, hence, you may pick whichever method you like.
In my code, Xpand code, the use of the Java extension looks something like this:

«IMPORT sdol» // import my model
«EXTENSION extensions::JavaSerialUID» // import the extension

«DEFINE genClass FOR Type»

abstract public class Generated«this.name» {
private static final long serialVersionUID = «this.generateSerialID()»L;
…

One impressive feature of the Xpand editor that you may take advantage of when debugging your extension (and of course when using it), is that the editor will parse the declared extensions and help you auto-complete. E.g.:

In the screen capture above, I’ve just typed ‘«'.
In the example below I'm using the pseudo-object-oriented option using the this as the object receiver:

Conclusion

In this blog entry I’ve shown how one may create an extension to Xpand using Java. This is a last resort option when using Xpand and Xtend, but not an uncommon one. The trick to building an extension is:

Create a Xtend file to expand the language

Must be in the class path where you use Xpand
A file with the extension ‘ext’
Place the file in any directory

Declare a mapping between a new Xpand function and the Java function you want invoked

Declaration of Xpand function in Xtend language
Declaration of Java in Java language (use Java packages, Java types, etc)
Use fully qualified names for model objects and Java implementations

Implement the Java function

Must be a static method
Can take any number of parameters
The first parameter may be used as this
Must be placed in the classpath where you use Xpand

Use the extension in your Xpand files

Import the extension using «EXTENSION»
Invoke the extension function using the pseudo-object-oriented or functional approach

Xtext Experience

2009-11-10T13:56:00.000-08:00

Over the last weeks I've been polishing a language for a client. As I mentioned before on this blog, I decided to use Xtext(www.eclipse.org/Xtext/) to build the toolset for the language. The language deliverables consists of:

An abstract syntax in EMF
A concrete syntax (in ANTLR originally, now in Xtext)
An Eclipse-based editor (textual)
A Model-2-Text (M2T) transformer that converts the input into Java code

This project has turned out to be a near perfect fit with Xtext. An for the first time in a long while, I'm very impressed. The Xtext team has done a fantastic job building a pragmatic toolset for building textual DSLs.

I did a similar language earlier for another client based on Microsoft's Oslo SDK. I was very impressed then too, but I actually think that open-source has managed to pull off a better (or at least as good) tool as that of M. Impressive!

In the process of learning Xtext, I did run into a few issues that with the help from google and the Xtext team I was able to resolve. In the next posts, I'll try to outline these issues and maybe also write a simple tutorial that highlights the issues I ran into.

M2T and Xtext

2009-10-30T10:19:00.000-07:00

I created a new language (again) for one of my clients this week. I love doing new languages and I’ve been working 20h a day on this one. I’ve been doing my last languages using the Microsoft tools, so I was afraid I would be a bit rusty on the open-source tools.

The language requires a textual syntax, so I decided to use ANTLR for the concrete syntax. EMF is of course my default choice for the abstract syntax.

After a few days I had completed the abstract and concrete syntax and had a pretty complete plan of how I wanted to perform the transformation. This is when I ran into some interesting choices.

First, I wanted to make sure I picked the proper transformation engine. The transformation is not very hard (it requires many transformers, but the distance from the abstract syntax to the finished products are not complex), so I decided for a direct model-to-text transformation (M2T). I usually try to avoid this as it makes the generation templates more complex than they need to be (that is, I like to do model-to-mode transformations until the transformation to text requires no logic), however, I’m not going to maintain this code and I did not want the client to have to also learn a M2M framework.

The M2T project at eclipse (http://www.eclipse.org/modeling/m2t) lists a few candidates. I was pretty sure I would select JET as the template engine. I already created a course section on JET (in our EMF course http://www.inferdata.com/training/java/emf.html), and I feel I know JET very well. It turns out that JET has taken leaps forward since I looked at it (and EMF does not take advantage of it yet), so I realized I’d have to spend some time learning the new constructs. I then went on to look at the alternative M2T engines. There are two listed in the M2T section:

Acceleo: http://wiki.eclipse.org/Acceleo
Xpand: http://wiki.eclipse.org/Xpand

After some quick evaluation, I could not find a reason to go from JET to any of these tools. Not that Acceleo and Xpand was not good, I just could not find any main reason to switch ideas.

Next, I wanted to also provide a good editor for the developers of my new DSL. I thought surely there has to be someone that has defined a code-generator with ANTLR as input and an Eclipse editor as output? I may have been sloppy, but I could not find one. Tempted as I was to write one, I realized, I will not have time… Then I ran across the Xtext project (http://www.eclipse.org/Xtext/). I’ve heard of it, but never fully investigated it. After my first investigation (and some communication with the main contributors, thanks guys!), I’m not pretty sure it is the right tool.

I’ll get back to the blog after I’ve finished my project, but it really looks as a brilliant piece of work. Xtext seems to be able to combine the ANTLR grammar (as concrete syntax) with ECORE models (as the abstract syntax). It also provides a very nice template based code generation language.

I’M ALL EXCITED!!!

C++, Oh Boy…

2009-08-19T20:31:00.000-07:00

This week I’m teaching a course on Advanced C++ in San Diego. I don’t use C++ on a daily basis anymore, but I still feel pretty confident… after all, I’ve been using the language since late 80’s.
I do sometime get stuck on syntax. It doesn’t bother me, as I know I can figure it out pretty quick.

As usual, I love the use of STL and how densely one can write relatively complex algorithms. However, every now and then, I wonder…

The one thing that got me this week was how to convert a standard string to lowercase. Having spent a lot of time lately writing code in languages where the goals of readability and maintainability prevails over performance and tightness, I have to admit I expected to see a upcase/downcase function on the std::string… OK, I should probably have remembered, but I didn’t…

I had to look this one up… Here is the way to downcase string in C++.

// assuming a string s
using namespace std;
transform(
s.begin(),
s.end(),
s.begin(),
bind1st(
mem_fun( &ctype::tolower ),
&use_facet<> >( loc )
)
);

To my students this week, I apologize for not dropping this in from the top of my head…
Seriously, I do see the argument that one should not add too many functions to the string, particularly if it can be done another way… however, converting strings to upper or lower case, is a very frequent task… Would it be too much to ask to add a ‘toupper’ or ‘tolower’ function to the string???

Design by Contract

2009-08-14T00:19:00.001-07:00

I was teaching a class that included a chapter on testing this week. I had added a section on design by contract and decided to look into what is available in the .NET space. I knew about Eiffel for .NET and some time ago looked into sharp#. I had to read up on the language specification of spec# so that I could add a reasonable contract for my examples.
I was very impressed with spec# and the static verification that Boogie provides.
I am even more excited that .NET 4 will provide a generic specification language. Maybe it is finally time for DbC to get out of academia and into mainstream?
For 15 years now, I’ve been teaching DbC at every occasion I could. I’ve long been convinced that it has significant positive impact on software quality. I’ve even had the chance to witness it close up the few times I’ve had success at convincing software organizations of DbC’s advantages.
So, why have DbC taken so long to catch on (assuming .NET 4 elevates it to main stream)? I have a few theories…
The first theory is that the DbC community seems to be constantly trying to show examples that are intellectually challenging to show how tight they can create a contract. These contracts become so complex that the specifications are significantly more complex than the algorithm required for the implementation. For a practitioner reading these specs, the concepts must look absolutely horrifying!
The other theory is that DbC often is presented in a confrontational style. I’m guilty of this too. I used to love to start my DbC presentations with statements like “I’ll show you something that will make you never write defensive code again”. Or I may focus on the fact that it is OK to crash and burn when preconditions are not met. Even though those statements may be theoretically correct, they will scare away practitioners. I promise I will never do that again.
No wonder, the community is apprehensive. Even academic giants like D. Firesmith end up writing an article showing how Defensive Programming is superior to DbC. Scary!!!
How about we as educators start focusing on the simple things that DbC can do… and assume that the contracts will be checked at runtime and that catching them will not make the system crash and burn… We can introduce the other factors later… when the concepts mature.
The statistics are greatly in our favor. Check the numbers from Lockheed’s experience from the C130J (Hercules II) project where the software quality improved by a factor 10 and productivity by factor 4… Those are strong numbers

Ruby on Rails

2009-07-20T14:30:00.000-07:00

I'd though I'd post something on the technology I've been most involved with lately... That is Ruby on Rails.
I think I've developed software on the most common web-platform. As of now, I think Ruby on Rails is the fastest (most agile) development platform.
My fluency now is such that I can pretty much keep up with the design of a site in real-time. That is, I can conduct a requirements session where I change/update the site while we discuss requirements. There are however areas where Ruby on Rails could improve. Here are a couple of things I'd like to contribute when I get some time.

Master-Detail editing template
Ruby on Rails allows me to easily create a CRUD web-site around a single class (usually mapped to a table). I can easily modify the web site to allow for the view/update of the dependent tables, but ONLY if the navigation is from the root-to-children.
It is typical, however, one edit a child with some associated meta-data/parents and then a set of details. Let me take an example. Say we want to create an application for keeping track of our music. We have a simple structure where we have the following types/relationships:
- Genre (metadata, we associate the albums with a genre)
- Artist (has many albums)
- Album (belongs to an artist, belongs to a genre, has many tracks)
- Track (belongs to an album, has many tracks
It would be easy to create a site where one first creates the artists, then associate the albums, and link it to one of a fixed set of genres, and then from albums we edit the tracks.
However, this is not really the way we would use such an application. A more natural use-case is that one has bought an album... We then want a form that allows us to enter the data of the album, associate it with an artist (or create a new artist if we don't have any other albums of that artist), link it to a genre (or create a new genre), then add the detailed tracks (the easy part).
To build such a site, is a project. Say I was doing one of my real-time requirements gathering sessions, I would have to save this one for the evening so that I can get a few hours of undisturbed time.
I want to extend the meta-protocol so that I can declare (as opposed to program) these editing templates. Or, I may just create a code-generation template (add it to the rake tasks for the invited :)).

Improve the error detection during code generation
There are a set of naming conventions in Ruby on Rails. The various concepts (like tables in the DB, controllers, Model objects, routing rules) are typically wired together by convention. That means that you need to know exactly when to use upper-case, camel-case, lower-case, underscore, etc. Usually, I don't make mistakes with this as I've used it quite a bit, however, when I teach Ruby on Rails, this is one of the most common mistakes done by my students. A bit of verification during code generation and elsewhere in the framework could give valuable feedback to the newbie's and make it easier to get started with Ruby on Rails.

Design Templates

2008-12-09T14:55:00.000-08:00

Today I started to write the design chapter for the book. I was trying to collect my thoughts on object-oriented design. The more I thought about it, the more hopeless the task seemed. I think it is impossible to define a single template that would apply to any software project. In my opinion, the approach to design, varies vastly across the different kinds of project.
If I were designing a real-time system, I would worry mostly about real time issues such as deadlocks, live locks, deadlines, etc. If I were designing a typical enterprise system, I would worry about very different things, say, service partitioning, database design, accessibility, security, etc.
So, I think what we’ll do is to provide a simple template in the book, and then expand the topic on a web-site.
I can see that I could develop many different templates, each specialized for a particular kind of system. Not that I think that there is a fixed template for any kind of system, but I do believe that such templates can be used as starting points for customized processes.
I think a template for design must have a set of optional features. Maybe I can create a template where a user can clearly see when to include a step.
For example, if your system contains data that must not get in the hands of outside agents, then include this “encrypt sensible field “step.
It will be an exciting task I’m sure :-)

Writing books...

2008-12-08T23:01:00.001-08:00

I'm currently working on a book together with Richard Mitchell and Vladimir Bacvanski. The topic of the book is how to model software... actually, I take that back... it is about how to describe domains with little or no ambiguity and then scope out a part that you want to solve using software systems... and then actually solve it...
It is actually more about the domain than the other parts... this because we're trying to keep the book at less than 200 pages!
The book is starting to take shape and I'm more and more optimistic that we'll get it out SOON!!!
I've been writing a chapter on system modeling today. My goal is to keep this chapter at less than 10 pages. It would be SOOOO easy to write 150 pages, but 10 pages seem incredibly hard. The good news is that whenever I'm done, Richard will probably rewrite it... and we'll have the usual poetry that only he can make :-)

About Domain Models

2008-12-08T23:00:00.001-08:00

At InferData, we always build domain models before progressing to system modeling. I've noticed that this is becoming more and more a trend in the industry although, what the various methodologies/methodologists define to be a domain model seems to vary.

We've been using domain models to understand a problem area. Our domain models are quite abstract, but they may be very precise.

In general, we define domain models using two aspects.

The first is a structural aspect. That is, we describe what the interesting objects are and how they relate to each other. The notation we use for this is UML class diagrams, although, we only use a limited part of the notation (Classes, Attributes, Associations and Inheritance).

The second aspect defines the dynamics of the domain (aka behavior). We describe the behavior as the effect that various events have on the objects. We describe the behavior using observational semantic (that is, we "imagine" that we have observed a change to the objects in the domain and we're describing this change. This is quite different from many other methods where the specifier thinks more like this "If I was to apply this action in the domain, the domain state will change this way").

It turns out that the observational semantic has several advantages. The most important of which (IMHO) is that it seems to be much easier to combine fragments of specifications.

This combinability (I wonder if that word will pass my spell checker :-)) makes it much easier to define and combine views.

I think the use of views in domain models may be a great thing moving forward. I can see us develop a set of orthogonal views that then can be combined into more complex models... More on that later...

SysML

2006-07-18T09:45:00.000-07:00

I just sat in on a presentation of SysML (http://www.sysml.org). Although the presenter did a good job defending the spec, I found the spec itself quite miserable. I am wondering if it is the fact that the specifiers decided to use UML Profiles as a method of specification as opposed to a completely new MOF model.
My general impression was that UML was overloaded to the extent of being useless as a graphical notation for communication. There are obviously some areas where the UML notation seems a good fit, however, for most views there seems that UML’s graphical notation confuses the issue rather than enhance it.
Some of my readers may know that there is a raging war between those that believe in UML profiles as a liable option for creating new specification languages and those that believe that every language should be designed from the ground up using a domain specific language tool (using MOF, EMF/GMF, or some proprietary commercial tool like MetaEdit or Microsoft’s DSL tool).
I’ve always felt that I would know when to use one or the other. There is of course a grey area where I would have to do some analysis back and forth, however, my selection criteria would be based on the following facts:
UML is a collection of graphical modeling techniques that when applied is commonly known to everyone and hence has the advantage that software engineers (and increasingly business analysts) can recognize the notation and quickly evaluate the specs in front of them. I’ve also noticed that having a common notation tends to trigger flaw detections based on pattern recognition
UML is limited in what it can express, hence, if I for instance wanted to show a spec of non functional requirements, temporal invariants, predicate rules, etc. UML would be an improper language. It would take too long to tweak the profile, the graphical language would be of little use (no advantages, rather confusion as it looks like something else)
If I have one of the tools that accepts UML profiles are available to the project I’m working on, I can usually leverage some key tool elements that would be hard to implement (e.g., visual comparison, version control, shared repositories, etc), hence, even if the profile was hard to define, the advantage of the tool vendors might make me chose a profile even though the language does not justify it.

After the presentation, I’ll have to go back and rethink my boundaries. I would have guessed prior to the presentation that SysML would be an excellent candidate for a UML profile. After all, describing a system is what most of the UML components were defined for, right?
Well… take a look and see what you think… Does UML give you better semantic definitions? Does the UML graphical language help you understand the underlying semantic? Is the language expressive enough? I’ll let you judge… My impression was, NO!!! A surprise to me.

Time flies...

2006-07-06T12:49:00.000-07:00

This week I’ve been working on a J2EE application for a client. I’m building a pipes and filter architecture for some batch processing (why you may ask as there are lots of open source implementaitons… well, the client’s have requirements that the open source (nor the commercial) implementations can handle).

I would usually always ensure that I test out my components carefully using unit tests. I did so here too (that is I tested out the filters carefully), however, the piping and process choreography required a JMS implementation, so I had to test my code in a deployed environment.

This turned out to be a nightmare. I found that my error-correction/time ratio dropped by about 100. It takes about 7 minutes to turn around from a code fix until I step through the debugger where I was before. Two things happen then. First, it steels time, second, the context I’ve built up by the time I found the previous bug has to be recreated.
Another fenonmen… because I always feel that I’m only this bug away from being complete, I kept on following this tedious process… well, two days later and still one bug away from completion, I realized that this could not go on.
Now I’m in the process of building a JMS/J2EE/JNDI simpulation/test environment.

I finished the parts I need to thest this code for the my client and the error correction rate is now back to normal…

Seems I can play Golf this weekend after all.

OORAM

2006-02-24T13:33:00.000-08:00

Almost 10 years ago, I was doing some work for a client when I found myself with some extra time on my hand. I had designed and implemented a real-time broker and I was waiting for the other development teams to finish up their work so that we could integrate.

A friend of mine that worked for Taskon had sent me a tool for the method OORAM and asked for my evaluation. I was pretty familiar with the method and had not long ago read (and I believed understood) the “Working with Objects – The OORAM Method” book from Trygve Reenskaug et al. Instead of waiting around, I thought it would be interesting to use the tool and the method to redesign the broker. It would be interesting to see if using the OORAM method could yield different ideas for my design.

When I read Reenskaug’s book, I liked it, but did not think of it as revolutionary. I had spent several years at Icon Computing where we worked on the Catalysis method; I had studied a slew of other design methods, and surely OORAM is just a slight variation…

I installed the tool and started to play with it with really no expectations. I was ultimately familiar with Design Patterns and all kinds of software architecture, having taught it for several years. Surely, no method or tool would help improve my PERFECT design :-).

Boy, was I in for a surprise… After a few hours in the OORAM tool, I started see a few abstract collaborations between roles in the broker that I had not seen before. After about a day, my view on the problem had changed completely. Then, mapping my new understanding in terms of roles, collaborations and various synthesis onto the implementation language (C++), I realized that I’d just come up with a far superior design. In fact, I was almost embarrassed by my previous design and spent the next few days reimplementing my broker.

Well, since then, Taskon (that made the tool) was bought up by another company and the tool was put to rest. The tool is no longer available to me and OORAM seems almost forgotten. What a shame!!!
A few weeks ago I was teaching a course on EMF/GMF, and I realized that with the current toolset in Eclipse, it would be relatively easy to build an OORAM tool for Eclipse. The metamodels are partially available from a submission from Reenskaug to UML 1.0 and I’ve even talked to Reenskaug about helping out on the project…

Now if I could only find some time to work on it (could someone please add a few more hours to the day???)…

By the way, if you are interested in the OORAM book, it seems to be out of print. However, you can download a PDF version (the last draft of the book before it went to the pubisher) here.

Pushing the Envelope

2006-02-22T22:18:00.000-08:00

About 5 years ago, I decided I would try to see how much of the Model Driven Architecture trend was hype and what was reality. Then (as now) I had been attending conferences where the claims seemed outrageous to me.
There were those that claimed that programmers soon would be obsolete. Soon computer software would magically appear from domain models and they would be so much better than what a programmer could produce (I even heard someone talk about how we could feed in natural language use-cases and the MDA tool that he was building would produce wonderful systems). I knew most of this talk of snake-oil-salesmen.
There were others that claimed that MDA was utterly impractical and that nothing good would come from it. The languages were already at its peek in terms of balancing expressiveness and efficiency.
I knew the truth had to be somewhere in-between the two extremes and having about 15 years of experience with generative techniques, I thought I would have a good chance of finding the boundary.
This finally led me to create a tool called MCC (I first called it MDAC, Model Driven Architecture Compiler, but the OMG lawyers did not like my name, so I changed it to Model Component Compiler). The goal has not been to sell the tool, rather to use it as a research bed to push the envelope of the Model Driven ideas.
I’m now on the fifth generation of architecture and I think I’ve been able to push the envelope as originally planned. The tool now encapsulates almost all the knowledge I have and what other people generously have offered as optimizations through the 5 years.
Anyway, the idea of this blog was not to talk about MCC but about pushing the envelope. I have for a long time kept a list of the techniques that I’ve learned by reading books and having experts criticizing the generated implementation. I’ve always found that I can encode the knowledge learned into the tool, but today I reread a book called Programming Perls by Jon Bentley and I found a nice optimization example that I’m pretty sure I’m not going to be able to formalize.

The book opens with a nice example where someone was asked to sort a rather large file of telephone numbers (all of them 800 numbers) when having only a few megabytes of main memory available (but plenty of disk space). The programmer also had severe performance constraint and was asking for Jon Bentley for help…

Now think about this for a second or two…. How do you do this? Which search algorithm? Merge sort? Quick Sort?
How do you split the files? How many batches?

Bentley came up with a really elegant solution. Because we know that all the numbers start with 1-800-, we can reduce it to its 7 digits. The presence of seven digit numbers can be expressed in a bit array where every possible 7 digit number is represented by a bit (hence, we need 10^7 bits, or if you want a bit more than 1 MB of memory).
We can now read the input file and set the bit for every number that we see, e.g., if we have the number 1-800-192-1234, we set the bit with offset 1921234. When we have read through the input file, we start reading off the bits from the proper end and for every bit set we write the corresponding number.

Now, HOW do you encode this kind of cleverness in your tool???

Divide et Impera or...

2006-02-14T19:51:00.000-08:00

Over the last weeks I’ve been working on writing a course on architectural thinking. I really looked forward to write this course so that I could organize my own thoughts on the subject.
There are of course a set of obvious topics covered such as TOGAF, Architectural Templates, Architectural Patterns, etc..., but the most interesting to me is the following topic
How does a good architect think?
I’m pretty sure I have a different mindset when I construct architecture from, let’s say, when I implement systems.
The biggest difference, I believe, is that in most other software related tasks I’m using some variation of divide and conquer approach. I may be dividing the problem into components, services, methods, classes, … I may be dividing horizontally, vertically, diagonally, … but the bottom line is that I’m taking a big problem and breaking it down into smaller more manageable pieces. Then I solve the pieces based on some dependency graph with some weighting based on risk/payoff.
We certainly do some degree of divide and conquer when breaking down architecture too, but I think there is an important difference in my thinking when constructing architecture.
The big difference is that I try to balance a set of different views/aspects and always try to evaluate my decision against all these aspects during the construction. Why is this important? Well, this is probably best illustrated with a little story told by a friend of mine.
He was working asked to review a newly designed architecture for a telephony company. The engineers proudly explained how you could with a click of a button redistribute new software to all their switches… “You just upload this file to here… run this command and then within seconds… all the nodes are running the new software…” My friend looked at it and then he asked, “… what is the security implication of this… what if one with malicious intent got into this directory and ran this command…” OOOOOOOOPPPPPSSSSS
The most constructive part of writing this architectural thinking course has been to identify all the views that I’ve forgot at some time in my life and to identify all the quality attributes I ought to have checked from all these perspectives. Now I’ll be all prepared to ask all the difficult questions for the next architecture I’ll construct.

Tightening the Envelope

2006-02-08T07:07:00.000-08:00

This week I’m again consulting and I’ve run into another issue with reflective system. At this client site we have multiple systems that all get their flexibility from use of configurable metaobjects.
The main issue to solve is how to overcome the dependencies between the various systems.
Unfortunately, there is my client is using different models for metaobjects across the systems. Different groups have different idea of what the metaobjects ought to look like (or if you want, they all use completely different meta-metamodels). In most cases, I think the metaobject have emerged organically with no oversight.
For those readers that have been in this predicament before, the issues are probably obvious, but for others… If my system is completely flexible and their system is completely flexible… then I have to understand how their system may change and build some kind of language for allowing mapping between metaobjects in the two systems.
My previous experience with this has been that if two systems build a model for metaobjects (or a metamodel), the two metamodels usually end up being very similar…
In this case though, they are not. The system we need to interact with has built a metamodel that is flexible but not very descriptive.
The other system basically consists of objects defined by metaobjects (that is every object is linked to its corresponding metaobject (e.g., “John Doe”: Object is linked to “Person”: Class). Each object can have a set of attributes, (e.g. “John Doe”: Object OWNS a “32”: Attribute) and these attributes are typed with metaobjects (“32”: Attribute is linked to “Age”: AttributeType). In addition every object can have a link to another object and the link is typed with a meta object. This model allows you to store information about almost everything, however, if what you store are at different meta levels, you run into some severe problems for the ones that access you… that is, what kind of object is this really? Does it represent an enumerated list? Is it a object with content? What is it…
The system I’m building has a much richer metalanguage allowing for definition of constraints, structure, enumerated types, etc…
I know a lot of theory about this topic, but so far I’ve come to the conclusion that my problem here impractical to solve… The model that was built for the other system does not provide sufficient information to build a proper map… This seems to be a recurring problem where flexibility is a retrofit…
A bit of advice: When building a metamodel, DON’T start from scratch. There are lots of metamodels around that are WELL THOUGH OUT and allows you to define ALL the rules required to constrain the objects without loosing flexibility or semantic…

Metadata

2006-01-23T14:44:00.000-08:00

Today I had a long discussion on metadata. I was arbitrating a discussion at a client site and the meeting was initially a mess. It turned out that everyone had their own definition of metadata.
The system we’re working on is a highly configurable distributed web-based content management system. The system is configurable to be able to adapt to any new kind of content and linking between the content.
The development team had been using the word metadata to define the data that defines the structures and editorial rules for the content (e.g., a person has a first name, last name, birthdata, etc).
Another team had used metadata to describe lifecycle information around the content (e.g., when created, original source, etc).
Yet another team used the term metadata to define the taxonomy of the data.
So… I did a search on the net to see if someone had a good definition we could ‘borrow’. After reading a few hundred definitions online, I’ve come to the conclusion NEVER to use the word metadata again without an accompanying glossary.
I’ll probably eventually end up using the definitions from W3C… that is, metadata is generic term that can mean all the definitions above and to be really useful have to be further qualified.
In case you’re interested, here are the various terms we’ll use:

Administrative metadata:
Metadata used in managing and administering information resources, e.g., location or donor information. Includes rights and access information,

Technical metadata:
Metadata used by design/implementation team to describe implementation lifecycles (e.g., logical delete).

Onthology Definitions:
Definition of data structures and semantic relationships.

Taxonomy Definitions:
Data required to support categorization and organization of content and metadata (typically for the purpose of navigation).

Blogging

2006-01-23T12:52:00.000-08:00

I’ve never done blogging before, so --- here is my first blog (I don’t actually know if this entry is called a blog or if blogging is a proper verb…).
Recently I’ve read a lot of interesting posting in blogs and now it is time for me to contribute too.
My intention is to provide a set of technical or philosophical insights and post them on the net.
Normally these insights end up in my notebook and are forever lost. A few days ago, a friend of my happen to browse one of these books (there are literally hundreds of them) and said some of the ideas might be worth publishing… I guess time will tell.