Puakma: Under the hood

• Puakma

I'm now officially over the Eclipse immediate IDE learning curve. Two nights ago I moved the Puakma project over to the Eclipse development platform. I knew there would be issues, but 118 source files and 3+ changes per file later we have a source tree with no compiler warnings. The most common warnings were unused variables/methods and imports that were never used. The changes were simple albeit very, very tedious. The good news is we're done :-)

I ran the javadoc creator and have discovered (a fact I was blatantly ignoring!) 265 javadoc errors/warnings. Now I need to set some quality time aside and work my way through updating the source file documentation. Should be fun. Not. Programmer l-o-v-e documentation.

The good news is all these pain in the @ss problems for me mean better, smaller, faster code for you. Can't be a bad thing.

I have also switched across to the latest version of the Vortex IDE (1.1 which is also based on Eclipse 3.2, hmmm what a coincidence...), which addresses a number of issues in Vortex 1.0. For me moving to 1.1 for my daily customer facing development is a huge step. It means it works well enough that I can get some serious customer work done. Of course, 1.1 is not perfect, but it is another step along the way to my idea of perfect, and each step builds on the last.

• Puakma

Enough talking about it. Now it's done. Today I have officially moved Puakma development on to Eclipse 3.2.1. A few days ago Martin released a stable enough version of Vortex 1.1 (based on Eclipse 3.2) which we have now switched to for development of web apps. Netbeans 3.6 is hell old and I have been waiting and waiting for a good version of eclipse before I made the leap. As expected it has taken me about 5 hours to port all the netbeans project setup across to Eclipse. The good news is Eclipse helps you write better code (eg by telling you what variables you've declared but never used), so I've been through all the source tree and tweaked the source to remove (almost) all the warnings (about 3 to go).

Strangest thing so far is setting up a jar archive. I like to have my source for editing, then click an icon in the project to generate a puakma.jar. Eclipse has a strange way of setting this up. It almost makes sense now but was difficult to initially work out. 

Tornado is across, now just ESSO and the Ltpa API to go, then it's bye-bye Netbeans 3.6.

• Misc

Has anyone else noticed a surge in spam over the last couple of weeks. I have got now >300 spams in the last 2 days! Not happy. I am guessing I now spend about 30 minutes per day cleaning the crap from my mailbox. 

I am busy trying to get kspam working on Linux without too much success :-( 

I now see these sneaky bastards are purposely sending mail to a non-existent address on my server with a from address of who they really want the mail to go to. Now it appears we are all acting as unwitting transfer agents. I have just turned off sending delivery failures - now they stay in the mail.box.

This is all such a HUGE waste of time.  

• Puakma

The next version of Tornado is nearing completion. We're just waiting a little to get 1.1 of Vortex a little more rounded out before the release so we can release both simultaneously. The next Tornado release has hugely updated system databases for better administration and web design through the browser. There are 101 internal tweaks fixes and updates too.

Vortex IDE 1.1 will be based on Eclipse 3.2 and have a ton of new internal functionality including a GUI database designer, included help system and better build control.

We currently have no firm release date, I'll keep annoying Martin until Vortex is ready ;-) 

• Puakma

Ok, the title is a bit misleading since anyone who knows me knows I LIVE for change. I find the day to day predictability droll and do what ever I can to liven it up.

Except when it comes to compilers.

Yup. You read it right. I still develop Tornado with Netbeans 3.6. According to this page , Netbeans 3.6 went gold on 13 April 2004 (trivia: coincidentally my grandfather's birthday). The important question here is why. Some may agree that Netbeans 3.6 is not be the perfect IDE and it's over three and a half years old to boot. Here's why I'm still using an antique compiler: Because it takes me too long to learn a new compiler and I lose too much productive time trying to work out where that feature I use all the time has been hidden in the new release. I would rather suffer through the compiler I know than face the pain of trying to learn something new.

This didn't all just happen by some accident. I went through JBuilder 1, 2, 3, 3.5. Netbeans 3, 3.5. Each change in compiler was a HUGE speed hump. You need to set up a new project in a new format with a new set of project options. Source files may need to be copied or moved. And you're not even sure that once you go through all this pain that you'll even like the new version! 

Having said all that I am planning a move to Eclipse in the near future (but no firm date). I do all my consulting work using our Vortex IDE running on top of the Eclipse framework and I like it, but more importantly, it helps me write better code. Eclipse does a lot of the precompilation work to tell you which parts of your source are not used, which imports need not be and so on. 

Fear the change : I think I'm getting old! 

• Misc

This last week I've had to chop and change between: Oracle, MySQL, PostgreSQL and HSQLDB. All good databases and all quite different. Some of the differences are a little annoying which makes it a challenge to write an application that works reliably across all platforms.

MySQL (on Linux) requires you to get the case right for table names, so in your code if you have a "SELECT * FROM TABLE" this is not the same (and will fail) on a "SELECT * FROM Table". This is because MySQL stores its data in a file or two for each table called "TABLE.MYD" and Linux being case sensitive means the OS can't find a file called Table.MYD. This approach is nice because backups and restores of a single table or database are easy.

PostgreSQL doesn't suffer the same case sensitivity issues because it stores all its data in a myriad of oddly named files in a bunch of directories. What postgres does choke on though is inserts. For example you have a table with an INTEGER field called age (but you wouldn't right? this is just an example). You use Tornado's TableManager object to insert the data: t.setField("Age", "34"); Hmmm here we go trying to store a String (which converts nicely to an int) into an INTEGER column in the table. Postgres will spit it right back. Mysql happily converts it on the fly and stores it in the table.

Another thing that annoys me is the LIMIT clause. This is not part of the SQL standard so vendors are free to implement this however they like.

Postgres: "SELECT * FROM TABLE LIMIT 10 OFFSET 32"

MySQL: "SELECT * FROM TABLE LIMIT 32,10"

HSQLDB: "SELECT LIMIT 32,10 * FROM TABLE"

Oracle:  "SELECT * FROM TABLE LIMIT 10" (no support for offset)

Microsoft SQL Server: Forget it, the method is too bizarre to discuss.

What amuses me more is if you pass a mysql formatted LIMIT clause to postgres, it throws and error and tells you exactly what you need to change for use on postgres. Hmm. If you're smart enough to be able to tell me what I did wrong, why not just swallow the LIMIT and do the work?? 

My primary goal when writing Tornado was to be able to abstract the database layer to the point where customers could swap out the database server with whatever they wanted to. While this goal has been mostly achieved, as a developer you need to be aware of these little idiosyncrasies in order to make it possible. 

• Misc

Here's a hot tip for anyone attempting a skypecast: DON'T.

Last night's inaugural Tornado Developers meeting went well, but the skypecast didn't. After wrestling with the time (making sure the cast would start when I expected it to) I finally got the skypecast ready for people to join. Unfortunately it appears in the public skype directory meaning EVERYONE on the planet can join in. BAD, VERY BAD. All these goobers from all over the world started connecting in and yelling and hollering into their microphones completely destroying the meeting. A little like having a bunch of rowdy gatecrashers overtake your party. The meeting was quickly moved.

I had a wee chat with Chris from podmatcher last night and we're going to work together to roll in the ajax chat system I wrote last week with podmatcher. I am optimistic this won't take too long and we can move our meetings permanently to podmatcher. That should save a huge number of headaches and make the chat easily accessible to everyone - even from your work, through a firewall :-)

• Puakma

3,001 Ajax web chat applications have been written. It's almost a right of passage for developers learning the capabilities of the Ajax paradigm. I have been putting off writing one because there were already 3,000 in existence, but last week I bit the bullet and scratched my itch. Sometimes it's good to do stuff just because you can.

The problem I had to solve is: We are about to start a weekly Tornado meeting. I want everyone to be able to find the chat session easily and have it start at a designated time (and finish so noone can rejoin an old one later!). Further, everyone shouldn't need a fancy schmancy chat client and an account somewhere (eg, skype, msn, icq, ...). 

So how does it work?

I have two tables, one called CHATEVENT which holds the meeting information such as start date, time, description etc. The other table is called CHATDATA and this holds the text people type, only row per chat message tagged to a CHATEVENT. At this stage I am not concerned about managing invitees but that could be easily added later. YAGNI 

On the client side, each time you type some text in the input field, it is sent to hte server via an ajax request and poked into the CHATDATA field. There is also a client side thread that polls the server for new messages, reading the CHATDATA table to find any new messages since the last poll. Any new messages are returned and dynamically added to the div which holds the chat messages, creating the illusion of messages being pushed to the client.

The trick I found was to ensure everyone only uses the server time for timestamping messages and have the server return the time of the last message. If you don't you'll get things doubled up - or missed.

On performance, because the ajax chat mechanism works on a polling basis you need to tweak the poll interval so that each client will poll regularly enough so the user experience is nice and infrequently enough to reduce the server load.

Hopefully we can get this integrated into podmatcher soon and start using it as the chat medium, that way if you miss a meeting you can come back later and read the pod meeting's transcript.

If you'd like a copy of this Tornado app, email me :-)