I know it's in bad taste to always reinvent the wheel, but you know what can I say. Initially when we began to work on EventMagnet, we were using Django. I found the opinionated nature of Django to be both limiting and not really in line with my opinions, so we moved on. I had some experience with TurboGears from working on TurboX2, and was not really fond of CherryPy style URL matching or SQLObject. So I decided to go it alone and put together my own framework for doing all this stuff.

When I first started writing the framework it was implemented as a CGI application using Routes to do URL matching. I really do not like regular expressions, so Django style URL matching was out, but I also didn't like the object style matching of CherryPy. Routes provided a system with a URL matching mechanism which was to my taste, although a bit weird. Routes weirdness comes from it being a direct clone of its Rails roots. The use of Ruby symbols in the URL seems like just funny syntax to a Python coder, and something fairly natural to a Rubyist. Funny syntax or not Routes is quite powerful. Just today I started to use the conditions feature of the URL Mapper. My biggest problem with Routes is really that the documentation for integrating it with a framework isn't that great or in depth. For example one needs to set the environ on the Mapper object to be able to use conditions but this isn't documented on the instructions for framework integrators page or in the Routes manual. Instead I had to dig through Routes code. Luckily it's very well commented and straight forward.

After I had URL matching sorted I had to deal with a way to do templates. As easy as it might be, using print to generate pages is never a good idea. :) I've gone through phases on what sort of templating system I like, I learned quickly while working on TurboX2 that I did not like kid, and by extension templating control embedded into XML style templating systems, at all. In a sort of reaction to that style of templating I first tried things like Django templates (which Guido has expressed love for) but I found myself annoyed that we were embedding code in the XML files. It was sort of the opposite of the problem with printing the XML ourselves. Instead of putting XML into our python we were putting a Python like language into our XML. I decided that this just wouldn't do, and went looking for another option.

So I came across PyMeld. Meld made the separation that I was looking for. Inserting our dynamic content was done directly in my Python code, without any XML that we might need to change later. This lets us localize anything aesthetic/structural to our XML templates and localize anything having to do with gathering and inserting our content in the code. Everyone has been happy with the position of Meld, but we've had some issues with its actual implementation. Many of the things that I've really hated I've fixed. For example in pure Meld you might do something like:

base = Meld("<a id='blah' href='url'><div id='href'>..</div></a>")
base.blah.href = 'http://127.0.0.1/'
base.blah = 'a different name'

I immediately saw this as a problem so I separated these two functionalities. I implemented structural access via __getattr__ and attribute (in the XML sense) access via __getitem__. I also renamed all the functions to be in snake_case, since javaCase makes my eyes hurt. Still Meld has some problems that are starting to drive me crazy. In particular anything you want to modify has to have an id attribute. Things like the title tag aren't supposed to have an id attribute, so this causes some problems. In light of that I've started to develop a new templating system with the same intentions as Meld which would also provide a DOM-like API. The DOM-like API will make it easier to access things without an id. I also intend to provide ways to generate new Elements similar to how MochiKit does it. I try to avoid doing Element generation in the code at all, but in some cases it's not possible to avoid that, so I would like to have a clean API for generating those elements instead of raw XML inlined. The DOM-like API is more or less complete, but nothing has been built up from there yet, and I had to set it aside to work on some other things.

I use SQLAlchemy for the database, I use Elixir because I'm lazy. In general my views on the state of database access in Python are that it sucks. In particular I hate the way querying is handled, as I've detailed here and here. To go further on that issue I'd like to say that I'd like transactions to be segregated into functions like they are using Blocks in ActiveRecord in Rails and like they are with Mnesia in Erlang. Of course it's not as easy with Python since it doesn't have multiline anonymous functions, and I believe I read that Guido had declared such a feature syntactically impossible. At least it's possible to do transactions with Python 2.5's with statement.

I did a lot of work today to clean up various ugly points in the framework. I added text for all of the errors so that an error page can be shown with appropriate descriptive text. I added a configuration file so I don't have to run through the source looking for references to the project I grabbed the copy of the framework from. I also made imports one level up possible so that I don't have to change every import line every time. I added the ability to return different types of data (JSON, XML, XHTML, etc) with a mix of Routes and Python magic. There are now Routes mappings for /xapi/:controller/:id, /api/:controller/:id, /:controller/:id, and /rss/:controller/:id. Each of these has a res_type variable (xml, json, html, and rss respectively). The __call__ method the Controller class then checks if res_type+'_'+action_name is a method of the class, and if it is calls it and if not returns a 404. So now I implement functions like json_list(), html_list(), and so on, and if I don't implement them there is just a 404. It works nicely and it makes my life a lot easier. I put all of the common code into a function called _action_name() and just call it from within the return_type functions.

On the JavaScript end we are also developing our own toolkit which is being built atop MochiKit. All the work is being done by Aaron, who has emphasized in his design not the creation of reusable widgets but instead reusable components that can be used to build widgets and interfaces. The main stay of Sparks is collection of DataTypes which are designed to be integrated into an interface and make extensive use of signaling, so operations like adding to a List emits a signal and the likes. This makes it much easier to produce rapidly changing interactive interfaces quickly since almost all of the event handling logic is already internal to Sparks and so it's just required to copy the objects and modify them as needed to get the desired behavior.

The days of CGI for the framework are incidentally over. It's now a WSGI application. I again opted to do it all on my own creating my own Request object and my own way of handling Responses. At this point I'll probably be changing the Response system to use an object instead of the more adhoc returning of a variable up to the top method that is currently in place, and proving to be bad. The Request object makes extensive use of property() to avoiding parsing information that we don't need (like cookies and various POST variable type stuff). Currently we handle input of either text/json or application/x-www-form-urlencoded, though the text/json support is a lot better since most of the API calls are designed around it. I will be using result type dispatching to provide application/x-www-form-urlencoded wrappers for the various API calls at some point.

There are both advantages and disadvantages to rolling your own framework, but it certainly teaches you a lot about how a web application request/response process works, and gives you a lot more control over your pieces. I personally really like to be able to control exactly what is used throughout the system, and so designing my own framework was very advantageous to me. On the other hand I spend a lot of time making it better instead of just accepting that it is limited, like I might have to do with an established framework, and working on my app instead. :)