my blog

On markup and markdown

Sat, 17 Mar 2007 07:38:20 GMT

Markup for all sorts of online systems is an annoying issue. It's difficult to find a compromise between the expressiveness of HTML and the simplicity of just writing in some plain text format. Luckily there are some big alternatives avaliable to simple HTML, but they're not exactly perfect either.

Using plain HTML is not without its security problems too: Cross-Site Scripting is a big problem when opting to use pure HTML for markup, since it becomes possible to inject nasty things like <SCRIPT> tags in everyday pages. So then you have to start parsing out certain HTML tags and only offering a subset, which is different from site to site. Seems like a pain, it is.

There are a ton of alternatives to just plain text or HTML. More and more popular these days are Lightweight markup language which give much of the expressive power of HTML but in a format that looks like how you might write an email. That's a really cool feature, considering that unformatted text can now look like what you mean almost as much as formatted text.

With the knowledge that I've decided to use markdown for my blog since it is reasonably simple and BSD licensed, some of the issues with these simple markup languages ought to be brought to light:

They're invaribly written as a set of regex. Regex are useful when trying to do one off find and replace operations, but writing a markup language on top of regex seems like a bad idea. Regex parsers are not fast. Python's seems to be especially not fast (which may have to do with that the parsing and compiling to bytecode is done in Python and then passed off to a _sre type to be actually executed). A reasonably sized file with a lot of markup takes a noticeable amount of time to parse using python-markdown. It really becomes not acceptable for use in a lot of situations without some kind of caching, and even then it seems problematic.
Some of these markup languages allow HTML anyway. This is a problem since it doesn't really alleviate the XSS problem if anyone can put HTML in anyway. Of course it is possible to use a regex (!!) to remove the HTML, python-markdown even includes a safemode which does this for you.
While the idea of creating a format that is intuitive because it is how you might markup an email or plain text file is a great one, if there are twenty markup languages out there you have to know that all format plain text just slightly differently, it becomes just as much a problem as just writing html.

So all of that being said I use markdown on this blog, and I use it in git-wiki (at least in some installations), but I'm not sure I would use it in a production site where scalability was a concern. I've been toying with producing a simple markup language that is reasonably easy to parse without using a huge set of regex, though I fear I'd just be adding to the intuitiveness problem.

On the on-disk-format

Fri, 16 Mar 2007 07:00:02 GMT

While more and more applications are written using a database backend, it seemed to me that that was a bit excessive for a simple blog, and that by storing the data either in a git repository or as flat files I would have a lot more room to play around with metadata and so forth. It also makes development easier by leaps and bounds.

The particulars of the implementation are fairly simple. There are two types of objects: Entries and Posts. An entry is a directory of posts. An entry directory probably looks something like:

some_internal_name/
some_internal_name/METADATA
some_internal_name/first_part.txt
some_internal_name/second_post.txt
some_internal_name/comments/
some_internal_name/comments/first_part.txt
some_internal_name/comments/second_post.txt

So basically, each directory includes a METADATA file which includes some entries which are formatted like:

TITLE: I am MetaEntry
AUTHORS: <some_user>
PUB_DATE: 1000000 -0700

This format is identical to the format in the preamble of a Post. After the METADATA file there are a bunch of Post files. a Post file, starts with a preamble with some metadata entries followed by a blank line, followed by the text of the entry. These metadata entries are merged with METADATA with the Post entries taking precedence. Though the TITLE entry in METADATA is used for the over all Entry title.

The comments format has not been implemented yet, so I won't discuss it.

The METADATA format supports a variety of statements which are used both for internal purposes and for displaying information on the Entries and Posts.

Future posts to this Entry will cover the METADATA format in more detail, as well as discussing the developing comments system.

Introduction to the blog

Fri, 16 Mar 2007 07:00:02 GMT

I finally broke down and decided it was time for me to keep a blog. I decided it might be cool to write my own blogging software to generate the blog. There were a number of factors I considered rather important, and a number that I decided to include because others considered them important.

Ability to write entries in vi, and simply push them out to the blog in some way. I decided to do this by having the blog be generated from some on-disk format (which I'll discuss in more detail in the future). In the future I intend to actually move the blog data into a git repository from which the entires will be generated.
Has to support some kind of markup for generating blog entries that looks good in plain text as well as formatted for the blog. Comments should also be covered by this language. (I haven't actually implemented this yet).
Support for serial entires, so that entries can be grouped together as part of a series, and so that they can inherit some metadata from the series itself. The logic for this is partially implemented, but I intend to make it considerable more complex and powerful in the future.
Written in Python! What can I say, I love Python!

I will follow up with more entries about the blogging software. They will likely be included in this meta entry, as a way to implement the series support :)

Making XMMS2 lazy accessible

Sat, 17 Mar 2007 09:30:34 GMT

I love XMMS2. It follows a design I like, and it's fun to use and work on, BUT sometimes I just want to listen to music. A normal process to "just" listen to a song goes something like:

> xmms2 clear
> xmms2 add [some file] # can also use mlib searchadd, or radd, etc
> xmms2 play

Whereas with mplayer it is just `mplayer [file]'. So I've been secretly using mplayer to listen to music a lot lately because I'm lazy. I decided that I ought to remedy this by writing a simple client for XMMS2 which takes a file or a collection spec (to be used via searchadd) as an argument and then creates a temporary playlist which it adds the song to, then plays. During the playback it should show a status display with some basic keybaord controls, and then it should clean up after it's done. Basically mimicing mplayer's behavior but giving the power of the medialib to find what you want to listen to, and using xmms2 to do the playback itself.

So I tossed together a client to do this. Right now it can only handle files or directories as arguments and the status display doesn't support any keyboard commands, but It's getting there. I'm thinking it might be better to merge it into xmms2cli and just extend the status command for the functionity I want. I imagine that it would be great for people wanting to use XMMS2 as a helper tool for other applications to play a music file.

XMMS2's Summer of Code

Mon, 19 Mar 2007 22:30:57 GMT

So XMMS2 is participating in the Summer of Code again! This year we have a lot of really cool project ideas. I'm especially excited about Service Clients, Visualisation, and any potential client development.

This year instead of participating as a student I will hopefully be mentoring a/some really cool project(s). I've been helping students out in our SoC specific channel: irc.freenode.net #xmms2-soc with trying to get their proposals in order as well as their conceptions about just how XMMS2 works. In particular a lot of them seem to be confused by our IPC mechanism. The fact that it isn't just plain text and that it supports some nice data types has certainly been surprising or confusing for some. So, xmmsipc is a binary IPC protocol that supports serialization of: arrays, dictionaries, dictionaries with source data (propdicts), unicode strings, bytestreams, (u)ints, floats, and even the collections structure (xmms_coll_t). This makes possible some really powerful communication between clients and xmms2d.

Beyond simply support support for some nice datatypes we also get by a combination of xmms2d, xmmsipc, and xmmsclient we get further cool features. Communication between clients and xmms2d isn't simply limited to one time request-response operations. We include two ways to get updated data. The first is broadcasts. In xmmsclient we provide a way to attach notifiers to ipc calls, so that when the response arrives a callback will be notified. This is the most common way to use xmmsclient. Broadcasts are ipc methods that you request once and get notified every time they occur. These are usually used for small pieces of data (often integers) like the currently playing song id, or the playback status. Each time one of these things change if the user has requested to receive the broadcast they get it. Sometimes when there is a lot more data, it comes a lot more often, or the client is more likely to disconnect it we use the second option: signals. Signals are like broadcasts except the decision to receive the data again has to be made every time, by restarting'' it. This is usually achieved by calling xmmsc_result_restart on the result object.

XMMS2's asynchronous design leads to a need for xmmsc_result_t, or our result object. Result objects are returned immediately by any request that has to communicate with the server via IPC, at which point the user typically has two options: wait'' for the response, or set a notifier on the result. Waiting on the result is often used in simple clients, it blocks the client until the result comes, so is not really suitable for a GUI client. To facilitate this synchronous process XMMS2 provides a xmmsc_result_wait function, which sets up a mini-event loop which waits for the IPC event until it arrives. After this the client can pull out the data they want from the result object. The other option which is much better for GUI clients is to make use of our notifier system. For this to work something has to be setup to monitor the file descriptors used by the ipc protocol to read and write data between the client and xmms2d. Once xmmsclient is attached into some kind of event loop like that we provide methods which can read from the file descriptor when there is data to read, and write out when there is data to write. So a notifier is called when the data to read reflects the method to which the notifier was attached. That way a separate function can be made to handle the data that has just arrived from the server.

So, hopefully this clarifies for students just how clients communicate via IPC and some of the capabilities and functionality provided by our IPC system when used by xmmsclient.

Web 2.0 and the Semantic Web

Tue, 27 Mar 2007 17:34:34 GMT

I work for a Web 2.0 startup. We're currently developing some pretty neat stuff. In particular we're working on a Presentation Slideshow type tool built around S5 (with S5's javascript being entirely rewritten to be more readable and extendable) called Perihelium, and a social event calendar based where users can add, search, and RSVP for various events in their area, called EventMagnet. Developing for the Web 2.0 age'' means we have to pay particular attention to the various trends that are going on right now; some of which are better then others.

AJAX - AJAX is taken to mean a lot of things, in the most literal sense our sites do not take advantage of AJAX at all, we do not use XML; instead we use JSON. JSON is more straight forward to parse, and to my eyes a lot easier to read. XML is very complex and powerful. For much of the data interchange being done on the internet it's just not necessary at all. There is more to AJAX then interchange formats though, and that's the idea of a dynamic interface. Mosty when you say AJAX this is what people think, interfaces that change when you click things instead of loading a new page. Using AJAX provides for some rather amazing capabilities. For example, type in a zip code and we can fill in your city and state. This does not stop there though, The use of AJAX lets us create an entire site driven by events. If someone else updates information about a venue we can update the page you are looking at. The power of AJAX can not be underestimated, and luckily it's backed by a powerful and cool programming language, javascript (I'll cover why javascript is cool and not horrible like it's popular to think in another issue).
Seperation of information and presentation - Seperating information and presentation makes it possible to lay information bare to the user without adding any implicit organization or styling aside from what is implied by the information. This is a really great thing because it makes it a lot easier to grok the information at glance without necessarily having to see it styled the way it was intended. There is another cool possibility with this though. It makes it a lot easier to theme things and provide various stylings for a site. S5 and by extension Perihelium take advantage of this fact by providing only a simple format for the actual presentation which is styled by some CSS and Javascript to provide the interactive part of the interface.
Microformats - Seperating information from presentation only goes so far. While it's nice to lay the data bare to the user and dress it up seperatly that can only go so far if the user (or worse) the computer doesn't know exactly what information is being laid bare to them. That's where microformats come in. Microformats provide a way to specify what exactly the various information on a page is, usually by use of HTML attirbutes (specifically class, rel, and rev). These little helpers make it possible to more easily scrape the information out of a page, or to style the information in a special way. A few Firefox plugins have started to do just that, collect and style information in microformats. Microformats may be the gateway into a real semantic web, in a way that is much more likely be adopted then something like the current proposals.
The look - The look is ugly. The wet floor effect that graces every single Web 2.0 logo looks cheap, the MASSIVE buttons don't make it any easier for me to click on, and sometimes the effects are just unnecessary and gratuitous. That being said it still looks a lot better then a lot of the designs of the past. We've taken some of what we like and adapted it into our own: light colors, thin lines, 90 degree angles (not that annoying rounded ones).

A lot of the ideas that are coming into place in the Web 2.0 world are ideas that have been expressed in the design of the Web since the beginning but have never quite come together. A while ago this spurned the concept of the semantic web which was the idea to encode information in XML/RDF (RDF to create inter-informational relations) such that information could be shared in a more computer useable way. These principals made their way into sites like MusicBrainz, which directly used the XML/RDF toolchain, but recently the MusicBrainz people have been pushing towards a more Web2.0 design. Their python bindings now access data via a Web Service instead of via RDF, and have become considerably easier to use.

Web 2.0 opens up the door for this kind of abstraction without making such radical abstractions from the way it has been done in the past they it stops them from being adopted. Granted sometimes radical changes are necessary or at least the best solution, but the Web 2.0 model, properly implemented, can provide almost if not all the capabilites of the Semantic Web model without as high an input level designed in implementing a tool around it.

P.S. I've setup HaloScan support for comments at least temporarily.

SQL Query Iterator/Generator

Wed, 28 Mar 2007 21:48:14 GMT

I've always really liked the idea of Object relational mappers. I just don't like writing SQL, I find it annoying and cumbersome and with some more complex situations I have to actually draw out the connections being made to do it correctly. On the otherhand objects make sense to me so why not apply fields to attributes, rows to objects, and tables to classes. It certainly works well enough for getting the information in the first place but doing queries for the information never seens to come across in an as elegent way.

SQLAlchemy tries to provide a powerful but pythonic way to make queries on an SQL database. SQLAlchemy is interesting in that mapping is more of a secondary feature and you'll often lay out your tables and then put mappers on them instead of mapping them via their class definition. They also avoid the ActiveRecord concept which is more common in other ORM. SQLAlchemy's mechanism for querying seems powerful but also a bit difficult to use and just doesn't feel right to me. It aims to be pythonic but it just seems messy because of being unable to overload operators in python. It also tends to focus on the table instead of the mapper definitions, so if you're used to working via the mapper having to work via the table can be confusing. Some examples of SQLAlchemy queries:

s = users.select(and_(users.c.age < 40, users.c.name != 'Mary'))
run(s)
s = users.select(or_(users.c.age < 40, users.c.name != 'Mary'))
run(s)

Notice that work is being done via the table.c.col instead of via MapperClass.col, and secondly that functions like and_() and or_() have been made. These functions are because SQLAlchemy can't overload these keywords. The inability to overload isn't all of the problem though. The way the statements are seems off. I propose that an ORM would be better served to overload the iterator/generator mechanism so that a query can be used directly in a for loop, or in a list-comprehension. Consider a similar set of queries:

for a in User.select(user.age < 40 and user.name != 'Mary'):
    ... do stuff ...
users = [ u for u in User.select(user.age < 40 and user.name != 'Mary') ]
users = [ u for u in User if u.age < 40 and u.name != 'Mary' ]

So these are three potentially implementable ways of doing SQL queries in a more Pythonic way. The first and second case have been similarly implemented by someone on ASPN with their recipe for Generator expressions for database requests. The implementation is both scary and impressive. To be able to deal with non-overloadable keywords he walks the actual parse tree. The problem is of course that it's not entirely clear how safe or even how fast that is. Still he provides a foundation for generating SQL on a generator. I believe it would be possible to implement the third case at least partially without having to walk the parse tree.

I've recently been playing to see just how this could be done, and realized that there is a reasonably easy way to get a taste'' for all the compare operators being called in the if part of the list comprehension. Here is the code I threw together to prove this to myself:

class Idea(object):
        def __init__(self):
                self.contains_val = None
                self.begin = True

        def __iter__(self):
                return self

        def __contains__(self, item):
                self.contains_val = item
                if not self.begin:
                        return True
                else:
                        return False

        def next(self):
                if self.begin and not self.contains_val:
                        return self
                elif self.begin:
                        self.begin = False
                        return self.contains_val

                raise StopIteration

This will return the value of the contains operation for an expression like: [ a for a in Idea() if 'blah' in a ]

So with that in mind it would be possible to generate at least simple queries and then have the iterator return the results for them and have the compare operators simply return True in all cases (since we already know the data is correct). The function calls to do the comparisons (even though they'd just return True) might increase the time this took, and it's not entirely clear how well it would be able to deal with and, not, and or (in which case it might be necessary to generate ball park'' queries which would be comperable, though that might not work in all cases). It's also possible that the same kind of parse tree walking could be done in this case.

Iterator/Generator based SQL query generation could potentially provide a way to do database queries in the various Python ORMs that is comfortable to anyone used to doing searches on lists and dictionaries, and is an area in which there should be further development.

What Javascript is and is not.

Thu, 29 Mar 2007 22:41:57 GMT

Being in the Web 2.0 "business'' certainly provides me with a lot experience looking at and fiddling with javascript code. In all this writing and reading I've started to notice some trends in development of javascript. In particular I have noticed a tendency for people to write javascript code in a way that is paradigmatically similar to the languages with which they are more comfortable. So I've decided to (with tounge planted firmly in cheek) detail what Javascript is NOT:

Javascript is NOT Python
Javascript is NOT Ruby
Javascript is (suprisingly) NOT Java
Javascript is NOT bad

All of these trends are fairly common and you can see each of them when reading code. Java is invoked often with the fear of using object literals. Very often you'll see code like.

var s = new String();
s = 'blah';

Whereas it's totally acceptable to skip the first and just set a variable equal to an object literal. This is also common with Arrays, and to a lesser degree with Objects.

The Ruby tendency has been cropping up a lot with the metaprogramming trend. A recent presentation has provided a metaprogramming example of implementing FormBehavior. The proposal also recommends that users "[not] fear eval()'' and "use with().'' Douglas Crockford discusses the issues inherit with both of these on his javascript site. Crockford, especially through his Y!DN video series, provides great introductions to a more Javascript way of thinking.

I have to admit: I'm a sucker for the Javascript is Python tendency. Javascript doesn't make this attitude particularly hard by providing a lot of the -isms I'm used to using in Python. Iteration of objects and arrays can be done very similarly to how one might do it in Python for example:

var nums = [1,2,3];
for (var a in nums) {
    var num = nums[a];
    print(num);
}

nums = [1, 2, 3]
for a in nums:
    print a

So certainly some of the paradigms I'm used to in Python presist in Javascript; this same example would be done by using the each method of the Enumerable class in Ruby, and so would take a different approach. What's interesting is that where the each method could be a bit cumbersome to implement in Python, it's very straight forward in Javascript:

Array.prototype.each = function (func) {
    for (var a in this) {
        func(this[a]);
    }
}

var list = [1,2,3];
list.each(function (a) { print(a); });

This not only highlights that javascript isn't quite Python, but it shows that it isn't really Ruby either, yet it simultaneously highlights the ability to do closures and anonymous functions which Ruby developers consider one of Ruby's greatest powers. This example actually shouldn't be used since it would mess up iteration of the object, but it demonstrates another of the great powers of JavaScript. Objects are not instances of classes. Objects are copies of other objects which supply a prototype of how an Object should be defined. In this way an instance of Array is a copy of Array which inherits the same prototype as the base Array object. This provides some really great implications for dynamicity. It's no wonder that the conventional dictionary was merged with the object in Javascript. It's particularly easy to make a new object in this way:

function obj () { this.list = [1,2,3]; }
obj.prototype = {show_list:function () { print (this.list); }};
var o1 = new obj();
var o2 = new obj();

Here I'm using a constructor, but it's actually possible to implement objects without using constructors which Crockford discusses in some of his advanced javascript lessons. Prototype-based object orientation, anonymous functions, and lexical scope, make javascript a really exciting language to work with that is not something one should try to meld into what they're used to working with but a language where the developer should embrace Javascript's power instead.

FreeBSD Ports PR patch extractor and tester

Fri, 30 Mar 2007 00:24:57 GMT

Earlier today on IRC we were discussing the idea of a system which would extract patches from newly submitted Problem Reports in FreeBSD's ports, and (attempt to) apply them to a checkout of the port against which they were made and then if the application works run a tinderbox build of the port to see if it compiles cleanly. The script would then followup on the problem report with the status and tinderbox logs (if appropiate).

As linimon@ pointed out we already have a number of the necessary components of this chain of events implemeneted. The new web gnats PR reader has very good patch extraction capabilities, and automating applying a patch and running a tinderbox build is reasonably easy (with some deference given to figuring out the appropriate level with which to apply the patch), and a way to post follow-ups to PRs.

A tool like this would probably be very useful since a lot of times I think that as ports committers we can be scared off by certain types of PRs which may very well be well written. Java PRs especially come to mind, as they often scare me off. I think it would without a doubt speed up the PR turn-over rate, and would also probably get a lot of ''scary'' PRs committed and closed faster.

As far as implementation, it doesn't seem incredibly hard, extract the latest patch (though this becomes harder when we get into issues of whether patches should be applied over or instead of previous patches), apply it (again a couple of issues can crop up here), tinderbox it (this part is probably the easiest), follow up with appropriate information (also fairly easy). Many of the tools that would be used in this setup (like the patch extractor) are written in Perl though, so I run away screaming at implementing this (that and ENOTIME), but I think it would be a worthwhile project to see done. If SoC applications weren't already over I'd suggest it as a potential SoC project since it's fairly confined.

Anyway, if someone is interested in doing this I'd be happy to discuss it, and help out where time and knowledge permit.

Database Iterators and Erlang's Query List Comprehenions

Fri, 30 Mar 2007 21:04:50 GMT

I've written about the idea of using List Comprehensions to do SQL queries in Python before. I've recently been learning about and fiddling with Erlang, and I ran into something quite amazing. Erlang includes a module called Query List Comprehensions (QLC). QLC seems to be exactly what I was wanting to see in Python, and in some ways even cooler. It is even possible to generate the equivalent of a view using rules with Erlang's QLC.

I think examples best highlight the power of this model:

[ P.name || P <- table(person) ] % Names for all people
[ P.name || P <- table(person), P.age > 30] % Names for all people over 30
[ P.name || P <- table(person), A <- table(address), P.id = A.person, 
  A.zip = 10001 ] % All people whose address is in zipcode 10001

Pythonists will see their familar List Comprehension with a slightly modified syntax, in Python they might look like:

[ p.name for p in table(person) ]
[ p.name for p in table(person) if p.age > 30 ]
[ p.name for p in table(person) for a in table(address) if p.id == a.person and a.zip == 10001 ]

Especially cool is that qlc is not specific to any one database engine but instead simply requires that 1-4 functions be implemented for any object that implements an iterator. QLC can either use a (slower) next() and test method where it iterates through every entry, or it can use a lookup_fun which implements a faster lookup method. This is where an SQL query generator could be placed, allowing for much faster lookups.

Rules are a very cool feature that makes it possible to implement views. The same place where a view would be considered useful it is possible to use a Rule. Rules can make for more terse syntax and even increase legability. Here is an example of a rule:

in_zipcode_10001(P, Person) :-
    P <- table(person),
    A <- table(address),
    A.zip = 10001,
    P.id = A.person.
[ P.name || P <- rule(in_zipcode_10001) ]
[ P.name || P <- rule(in_zipcode_10001), P.age > 30 ]

I certainly won't be turning in my Python badge anytime soon, but I'm really impressed with some of the database querying capabilites presented in Erlang, and I think it's now more important then ever to try to implement similar functionality in the Python ORM world.

Yet Another Python Web Framework

Sun, 01 Apr 2007 02:18:57 GMT

I know it's in bad taste to always reinvent the wheel, but you know what can I say. Initially when we began to work on EventMagnet, we were using Django. I found the opinionated nature of Django to be both limiting and not really in line with my opinions, so we moved on. I had some experience with TurboGears from working on TurboX2, and was not really fond of CherryPy style URL matching or SQLObject. So I decided to go it alone and put together my own framework for doing all this stuff.

When I first started writing the framework it was implemented as a CGI application using Routes to do URL matching. I really do not like regular expressions, so Django style URL matching was out, but I also didn't like the object style matching of CherryPy. Routes provided a system with a URL matching mechanism which was to my taste, although a bit weird. Routes weirdness comes from it being a direct clone of its Rails roots. The use of Ruby symbols in the URL seems like just funny syntax to a Python coder, and something fairly natural to a Rubyist. Funny syntax or not Routes is quite powerful. Just today I started to use the conditions feature of the URL Mapper. My biggest problem with Routes is really that the documentation for integrating it with a framework isn't that great or in depth. For example one needs to set the environ on the Mapper object to be able to use conditions but this isn't documented on the instructions for framework integrators page or in the Routes manual. Instead I had to dig through Routes code. Luckily it's very well commented and straight forward.

After I had URL matching sorted I had to deal with a way to do templates. As easy as it might be, using print to generate pages is never a good idea. :) I've gone through phases on what sort of templating system I like, I learned quickly while working on TurboX2 that I did not like kid, and by extension templating control embedded into XML style templating systems, at all. In a sort of reaction to that style of templating I first tried things like Django templates (which Guido has expressed love for) but I found myself annoyed that we were embedding code in the XML files. It was sort of the opposite of the problem with printing the XML ourselves. Instead of putting XML into our python we were putting a Python like language into our XML. I decided that this just wouldn't do, and went looking for another option.

So I came across PyMeld. Meld made the separation that I was looking for. Inserting our dynamic content was done directly in my Python code, without any XML that we might need to change later. This lets us localize anything aesthetic/structural to our XML templates and localize anything having to do with gathering and inserting our content in the code. Everyone has been happy with the position of Meld, but we've had some issues with its actual implementation. Many of the things that I've really hated I've fixed. For example in pure Meld you might do something like:

base = Meld("<a id='blah' href='url'><div id='href'>..</div></a>")
base.blah.href = 'http://127.0.0.1/'
base.blah = 'a different name'

I immediately saw this as a problem so I separated these two functionalities. I implemented structural access via __getattr__ and attribute (in the XML sense) access via __getitem__. I also renamed all the functions to be in snake_case, since javaCase makes my eyes hurt. Still Meld has some problems that are starting to drive me crazy. In particular anything you want to modify has to have an id attribute. Things like the title tag aren't supposed to have an id attribute, so this causes some problems. In light of that I've started to develop a new templating system with the same intentions as Meld which would also provide a DOM-like API. The DOM-like API will make it easier to access things without an id. I also intend to provide ways to generate new Elements similar to how MochiKit does it. I try to avoid doing Element generation in the code at all, but in some cases it's not possible to avoid that, so I would like to have a clean API for generating those elements instead of raw XML inlined. The DOM-like API is more or less complete, but nothing has been built up from there yet, and I had to set it aside to work on some other things.

I use SQLAlchemy for the database, I use Elixir because I'm lazy. In general my views on the state of database access in Python are that it sucks. In particular I hate the way querying is handled, as I've detailed here and here. To go further on that issue I'd like to say that I'd like transactions to be segregated into functions like they are using Blocks in ActiveRecord in Rails and like they are with Mnesia in Erlang. Of course it's not as easy with Python since it doesn't have multiline anonymous functions, and I believe I read that Guido had declared such a feature syntactically impossible. At least it's possible to do transactions with Python 2.5's with statement.

I did a lot of work today to clean up various ugly points in the framework. I added text for all of the errors so that an error page can be shown with appropriate descriptive text. I added a configuration file so I don't have to run through the source looking for references to the project I grabbed the copy of the framework from. I also made imports one level up possible so that I don't have to change every import line every time. I added the ability to return different types of data (JSON, XML, XHTML, etc) with a mix of Routes and Python magic. There are now Routes mappings for /xapi/:controller/:id, /api/:controller/:id, /:controller/:id, and /rss/:controller/:id. Each of these has a res_type variable (xml, json, html, and rss respectively). The __call__ method the Controller class then checks if res_type+'_'+action_name is a method of the class, and if it is calls it and if not returns a 404. So now I implement functions like json_list(), html_list(), and so on, and if I don't implement them there is just a 404. It works nicely and it makes my life a lot easier. I put all of the common code into a function called _action_name() and just call it from within the return_type functions.

On the JavaScript end we are also developing our own toolkit which is being built atop MochiKit. All the work is being done by Aaron, who has emphasized in his design not the creation of reusable widgets but instead reusable components that can be used to build widgets and interfaces. The main stay of Sparks is collection of DataTypes which are designed to be integrated into an interface and make extensive use of signaling, so operations like adding to a List emits a signal and the likes. This makes it much easier to produce rapidly changing interactive interfaces quickly since almost all of the event handling logic is already internal to Sparks and so it's just required to copy the objects and modify them as needed to get the desired behavior.

The days of CGI for the framework are incidentally over. It's now a WSGI application. I again opted to do it all on my own creating my own Request object and my own way of handling Responses. At this point I'll probably be changing the Response system to use an object instead of the more adhoc returning of a variable up to the top method that is currently in place, and proving to be bad. The Request object makes extensive use of property() to avoiding parsing information that we don't need (like cookies and various POST variable type stuff). Currently we handle input of either text/json or application/x-www-form-urlencoded, though the text/json support is a lot better since most of the API calls are designed around it. I will be using result type dispatching to provide application/x-www-form-urlencoded wrappers for the various API calls at some point.

There are both advantages and disadvantages to rolling your own framework, but it certainly teaches you a lot about how a web application request/response process works, and gives you a lot more control over your pieces. I personally really like to be able to control exactly what is used throughout the system, and so designing my own framework was very advantageous to me. On the other hand I spend a lot of time making it better instead of just accepting that it is limited, like I might have to do with an established framework, and working on my app instead. :)

The problem with templating systems

Mon, 09 Apr 2007 19:29:08 GMT

In the past I've discussed my use of the PyMeld templating system. Meld is rather neat in that it converts an html string into an object model which can then be manipulated like one might manipulate any tree in Python. This has a number of upshots, mainly related to the seperation of code and presentation. In this case I have only 100% valid XHTML that isn't contaminated at all with a templating language that is either plaintext/comments or attributes embedded into the XML. Meld on the otherhand uses the id attribute to build a tree, but the tree is not strict, in that you can access an any-depth tree member from the top of the tree by id. This has some problems in that a) everything has to have id attributes, and b) id name conflicts can become a serious problem.

Meld's implemented by basically using a regular expression to find matching IDs in an HTML string and then inserting/overriding those portions of the string. For a large template this is a rather expensive operation. Even for small templates it has issues. I finally hit a situation where I had to give up on Meld.

So I was working on this site to do TV listings, and like usual I used Meld to generate my templates. At first the code was doing the RSS parsing and fetching on every run, and it was a bit slow, so I figured when I moved the RSS data into a database the site would get a lot faster. I was quite wrong. Meld was taking 3-5 seconds per episode to generate the entries! With only 15 entries on the page that was clearly not going to work, so I went to find another solution.

In the past I'd started to write a system that mimiced the DOM api but was a bit easier to work with. In about an hour I tossed together an HTMLParser setup that parsed the HTML and Text elements into a tree that could then be searched through using some DOM like functions. Right now it's a bit harder to work with since I have to create DataElement and HTMLElements when I want to insert things instead of just inserting text directly like Meld allowed. I'm hoping to make this process a bit easier. I'll also be merging the base class with the HTMLElement class since they do more or less the same thing.

I'm tenativly calling it Kiln and it's able to template by TV listings site fast enough that I don't notice it as being a problem. I do hope to sit down and make it really fast in the future though, and I'd really like to see this more manipulative type of templating become popular. I think that moving the code out of the template is a really good thing, since it seems one of the intiial reasons to use templates was to move the HTML out of code, and yet in response we just seem to have done the opposite!

Unified types and classes in Python are one of its most powerful features

Tue, 10 Apr 2007 15:45:20 GMT

Around Python 2.2, before I had started to code in Python, PEP 252 and 253 paved the way for what I consider some of Python's most powerful features. There are a few issues I'd like to focus on the first of those is the ability subclass python's base types.

So say we need a dictionary that when a key is set to a value on that dictionary the key is appended to a list instead of overriding the old value. This is useful in for example HTTP headers where things like Set-Cookie will appear multiple times but with different values. We have a few options to go with here. We can roll our own class which implements the same methods as dict and so would thus work just like a dict (aside from the changes we want). So what exactly would have to be implemented to make that possible on a reasonably usable level; we'll need: __getitem__, __repr__, __setitem__, __delitem__, items, iteritems, values, itervalues, keys, iterkeys, __contains__, and get. I'd say that would provide a reasonably usable and introspectively capable implementation of dict which we could adapt for our purposes. That's a lot of code that acts just like dict. So we have another option. We can use UserDict which is basically dict reimplemented so it can be inherited from which is back from the days before the great unification. So that brings us to the awesome approach that is provided by the grand unification. We can just inherit from dict and override the parts we need to change. Our example can look something like:

class ResponseHeaders(dict):
    def __setitem__(self, item, val):
        if item in self:
            iv = self[item]
            if isinstance(iv, list):
                iv.append(val)
            else:
                iv = [iv, val]
                super(ResponseHeader, self).__setitem__(item, iv)
        else:
            super(ResponseHeader, self).__setitem__(item, val)

    def items(self):
        ret = []
        for k,v in super(ResponseHeader, self).items():
            if isinstance(v, list):
                ret.extend([ (k, a) for a in v ])
            else:
                ret.append((k, v))
        return ret

    def iteritems(self):
        return iter(self.items())

That's a really cool feature that makes my life a lot easier in that if I'm using a dict to store things and I pretty much need a dict, I can adapt the dict to be just right instead of writing a new kind of object. The great unification had some other really cool side-effects though. It's now possible to override __new__ which is the method used to actually create the object instance in a class. A classic reason you might need this is for the Singleton pattern. You should of course remember the reasons not to use the Singleton. Anyway, it's good as an exercise to show off new:

class Singleton(object):
    _single = None
    def __new__(typ, *args, *kwargs):
        if not typ._single:
            typ._single = object.__new__(typ, *args, **kwargs)
        return typ._single

It might be better to directly call the object's __new__ method instead of calling object's __new__ method indirectly, but the power of being able to override __new__ is still quite clear.

I use this capability a lot. TagFu is built around overriding dict and list so that they read and write from a sqlite database instead of in memory storage. Combined with Python's expressiveness the capability of python's native types and the ease of overriding and adapting them are why I am such an avid Python programmer.

Summer of Code 2007 - XMMS2 and FreeBSD

Fri, 13 Apr 2007 18:43:52 GMT

This year's Summer of Code is off to an awesome start! XMMS2 got a lot of amazing projects (and amazing proposals to begin with) this year. I will be mentoring Florian Ragwitz this year in his project to implement a testing framework for XMMS2. Florian had a really great idea in that he plans to make the framework a language-agnostic as possible with the common factor being a unified output format, in this case TAP. We're really excited about this because it will making writing tests as painless as possible, since everyone will be able to write them in their favorite language. It also makes testing things like language-bindings fit more seamlessly into our overall testing framework. Florian will also be working to implement a sandbox for running xmms2d as well as test-case media libraries and other good tools for writing better tests. I'm very excited to mentor this project, and I expect that Florian will provide XMMS2 with an amazing testing framework this summer!

The other projects XMMS2 was alloted slots for this summer are also really exciting. Johannes will be finally re-enabling visualization and he has some great ideas to make this both fast and portable as well as network transparent, by utilizing different transport mechanisms depending on where the client is located and on what platform xmms2d is running. Thomas will be working Autogenerated IPC which should make it much easier to right and get full coverage for XMMS2 bindings. This should also make coverage a lot more consistent, so we're very excited to see this kind of normalization being implemented for XMMS2. Finally Ning Shi will be working on Service Clients. this is an idea that theefer and I have tossed around for a while which will allow clients to implement method that other clients can execute with xmms2d mediating the communication between the two clients. We think this will allow for a lot more powerful features in every client, and will also reduce code replication since fancy features like MusicBrainz querying can be done by a service client instead of each client having to implement it on their own. Good luck to all of the students, we're really excited about this year!

On the FreeBSD side, I'm totally thrilled about some of the projects going on this summer! There is a big focus on performance this year, which in light of jeff@'s recent graphs is especially rewarding and exciting. I'm also happy to see a lot of work going on in the ports domain including the promise of parallelization which would probably really help on pointyhat and with all those new fancy dual core computers (anyone wanna send me one? :D). The one thing I was a little confused by was yet another installer project. I have no doubt that Ivan will do an awesome job but it seems like we have not one but two incomplete attempts to do this very thing, and I'm not clear why someone didn't try to pick up where those left off instead of trying to do it for a third time. Anyway, a new installer wouldn't hurt anyone so if this is what it takes then that's not so bad.

Again, good luck to all students, and let the games begin.

WebKit on FreeBSD

Thu, 19 Apr 2007 22:58:15 GMT

Today I decided I should give WebKit a try. WebKit seems really neat and it would be great to have an alternative to Firefox that isn't Desktop Enviornment bound. I got it to compile aside from a couple of issues, the folks in #webkit on freenode are super helpful and quick to respond. I even got one of the fixes I had to make committed while I was still working on getting WebKit to compile. :)

WebKitQT uses QT4 which is really a pretty cool framework for doing GUI app development, and it makes it possible to do some really neat effects, and tricks. One of them is evident in the Test QT browser that comes with WebKit. A progress bar is overlaid ontop of a page as the page loads, which disappears once you start to scroll. I thought it was pretty neat. On top of that the test browser is really fast, especially in light of some of the issues I'll go into in a bit. The rendering also looks really nice, the fonts look better to my eyes then they do in firefox.

Anyway, I ran into some issues and so I'd like to both inform people of them and see if anyone with some FreeBSD threading-fu can lend their brain:

Firstly, I had to fix it to set a define so it would call pthread_attr_get_np() instead of the Linux equivalent. I later ran into a Bus error which I found was fixable by calling pthread_attr_init() on the attr struct before trying to set it with pthread_attr_get_np(). I did this since the manpage said it was HIGHLY RECOMMENDED and it did fix the problem which was causing javascript to crash the test browser
It doesn't run outside of gdb. If I try to run it outside of gdb it will eat about 90% of my cpu in state ksesig without ever loading a browser window. I've put the thread apply all bt here. This is a backtrace from when I start the test browser outside of gdb and then attach gdb to the running process. I'm really not sure what's going on here, and it's further annoying that it works just fine if started up inside gdb. Anyone got some insight into what FreeBSD might be doing in this case? I'm using FreeBSD 6.2-STABLE from Sat Feb 3 22:51:13 PST 2007.

Anyway, WebKitQT is really neat, and I'd really like to see it get solid FreeBSD support so that when it's released for real I can add a port that actually works and doesn't require much or any patching. :)

Results of Test Run

Wed, 02 May 2007 00:12:51 GMT

I finished up the test run a couple of days ago. It went fairly well, though I will probably clean up the script some and rerun it (the script needs better mechanisms for handling fetch fails). The results of the run can be found here. The summarized version looks something like this:

[0:02] alex@Laptop: ~/fetches> cat output.tap |grep -c "unzip failed"
49
[0:02] alex@Laptop: ~/fetches> cat output.tap | grep -c "bsdtar failed"
4
[0:04] alex@Laptop: ~/fetches> cat output.tap | grep -c "unzipped results did not match"
4

For those playing at home, the first case means the distfile was bad, which can be remidied by implementing distinfo checking code in the test runner. The second case means bsdtar was unable to decompress the archive (these are bad news), the third case means the results of the two decompress operations didn't match (this is probably the worst news). Once I fix up the script to handle distinfo and report better information (I stupidly forgot to put the port name in the test description!) I will go and test the not ok results by hand.

In light of the eight failures I will probably when submitting the patch to switch to bsdtar for USE_ZIP also add a USE_UNZIP_FALLBACK knob incases where unzip is required.

Switching from unzip to bsdtar in FreeBSD Ports

Mon, 30 Apr 2007 01:15:40 GMT

FreeBSD is very lucky to have an implementation of tar based on libarchive. This combination has proven to be very fast and is under constant development, kientzle@ has and continues to produce an amazing piece of software.

Perhaps one of libarchive's best kept secrets is its format support which includes tar (of course), cpio, iso9660 (!), zip and now even ar. I dream of seeing rar and write support for iso9660 but that's for another time. Anyway, the fact that libarchive supports zip made me wonder why it wasn't used in Ports. After discussing it briefly on IRC I came to the conclusion that there were fears of regression since zip is not exactly the best defined file format. With that in mind I decided to write a test runner which would fetch every distfile that used zip in the Ports collection and test whether it:

uncompressed with unzip
uncompressed with bsdtar
whether the output of unzip and bsdtar were identical

Once I extracted the distfiles list (which I did using a shell script to traverse the ports tree and grep for USE_ZIP, followed by a python script which parsed the output of that and the output of make fetch-list into a dictionary of lists of distfiles), I wrote up a test runner to do the above test case and I'm currently running it. So far it has gone through 26 of 543 ports with two failures that were the result of bad distfiles (I didn't implement distinfo check and retry in the test runner). I will test all the failures by hand once it is done and then if all goes well I will work on a patch to make bsdtar the default tool for doing unzip in the Ports collection!

If anyone is curious I can supply a pickled copy of the distfiles dictionary and the test runner script.

Repository format and workflow diagrams

Wed, 09 May 2007 04:27:36 GMT

I decided to call it dbugt, and I've been planning out just how it would work a lot lately. I decided that it would be important to describe how the distributed approach would be used in actual bug support workflows and so I've created diagrams of branching, and merging to show some situations where common distributed paradigms would come in handy. I'm of the opinion that backing this with a regular old version control filesystem would not be a suitable approach since merging would be difficult if a bug were stored as a single file and if we end up breaking the bug into a number of atomic files, we've basically implemented a repository format inside another repository, so it seems like a better idea to just pull that repository out into its own.

The Repository Format breaks the repository into three distinct objects, Bug objects which include a collection of change objects, and change objects which are a tree of Atoms which describe specific changes to the Bug metadata. Each Atom is a single unit of change, in this way it's easier to merge and handle these repository changes. All objects in the repository are addressed by crytographic hash and include referential markers in their body so that the Bug graph can be built from the Bug data alone.

The Repository format tree might look something like this.

  Bug_ID
 /
|
|                      timestamp -> time/tz
|                     /
|           change_1-<
|          /          \
 \        /            owner -> alexbl
  Bug_ID-<
 /        \
|          \
|           \
|            \            status -> resolved
|             \          /
|              change_2-<
|                        \
|                         note_add: fixed!
|
 \
  Bug_ID

Branch situation: I had discussed briefly the prospect of resolving conflicts by branching instead of merging and conflict resolution. I've now had some more time to flesh that idea out and I've diagramed and described a situation where that might happen:

Change A: submitted at 12pm (bug initialization)
Change B: submitted at 2pm added at 2pm
Change C: submitted at 3pm added at 1pm

             submitter -> alexbl
            /
         A-<---timestamp -> 1178707906/-0700
        /   \
Bug_ID-<     summary -> ENOTIME
        \
         *
         /\      owner -> alexbl
       *  |     /
      /   *--B-<
      |   |     \
      |   |      status -> analyzed
      |   |
      |   |   owner -> fred
      |   |  /
      |   C-<
      \      \
       *------note_add: I think this is only on Platform X

What we have here is a new bug which has branched in two directions. A bug was submitted which two actors opted to work on, so the bug was branched into two seperate trees. When Change C is submitted the repository would detect a conflict and either automatically resolve it by a forced branch, or in this case the bug could be resolved with an interactive branch. The user was asked if a branch was an appropriate action, and after the user decided that it was they were asked if the non-conflicting change operations were appropriate to be rebased on to the first branch. Since the user felt the note might be useful to the person that submitted change B he opted to rebase the note onto the other branch. Note that the rebase occurs at the timestamp of the addition, so that the note was added before the B changeset was submitted.

Branching makes a very important operation in the dbugt because it allows users to work independently (and often times exclusivly) on a bug in situations where two developers might have different approaches to a problem, or even in situations where developers feel that a bug was closed without actually being resolved appropriately they can simply branch the bug and work on it themselves without creating a change war. Merge strartegies are highly interactive attempting to rebase as much pertinent information onto the other branches as possible.

Of course branching is not always desireable and certainly something that needs to be reversable so changes made on a branch can be merged into the master branch either entirely or interactivly, and the merging branch will be removed. This would be useful in situations where a bug had been branched and one of the developers resolved the bug and commited a fix, a developer could easily merge their branch back into the resolving branch so that information that was accumulated in their branch that had not already been rebased could be made public record on the unified bug report.

Another situation where merging is useful is for duplicate bug reports:

 Bug_1-change1--change2
/                     \
*                      \
\                       \
 Bug_2-change1-change2---*-change3

Here we have two bugs both of which have had changes made to them beyond the initial operations, Bug_1 had a change in its seocnd changeset which marked it a duplicate of Bug_2, at which point dbugt merged change1 and change2 into bug_2. Conflicts were either overridden, discarded or branched based on user input.

Ideas for a distributed bug tracking system

Tue, 08 May 2007 03:27:05 GMT

Having seen a couple of distributed bug tracking projects: DisTract and DITrack and having read this blog post. I decided it was finally time to put down some of the ideas I've had for a similar system. I decided to focus on a somewhat hybridized system so that one can for example submit bugs directly to a central repository, and not be required to have a complete copy of the repository local. The other major focus is the idea of having a unified interchange format which will be used to transmit bugs between two repositories who can then locally store the bug in whatever way they want. All changes to a bug will be atomized so that it is easier to do merges of the bugs by simply inserting the atomic changes into the history. I'm also curious about playing with the idea of implicit branching of a bug so that if two users submit that they are now working on a bug it will break the bug into two distinct branches where the two users diverged. Certain changes would propogate to both reports and other changes would propogate to only one of the branches. I've written down some notes about how the system will work and I'd like to hear from anyone that might have some ideas to make it better (or some suggestions on what I might have wrong):

Three components:

output
interchange
storage

Interchange is a format that all implementations understand it has two versions a binary and a text version. Interchange is available in both a diff and a full format. Interchange is read in by the client and converted to storage. Storage is any method for storing the atomic changes of a bug operation. Output is a mechanism for displaying the bug information. It can also be used to serialize interchange for clients where implementing interchange is not a suitable option. For example a JSON output might exist which would be used for getting the bugs into javascript instead of implementing an interchange parser in Javascript.

A user will add a bug at which point it will be written to storage, they may then submit bugs to a central bug repository or to another user. The bug is then serialized into interchange and pushed to the other store where it is deserialized and put into storage.

Users are not required to have an entire copy of the bug repository in order to recieve and submit bugs. Nor are they required to use the same storage mechanism as any other repositories they are working with. This way a central server can host the entire repository in a heavy weight database while developers can checkout only bugs related to their personal area of development and store them on the file system.

Adapters are specialized clients which take some sort of parseable format in and convert it to inter- change to be submitted to the central repository. This is used by both the web client and the email client who parse the input into a format which can be submitted. Command line tools also work by con- verting from command line arguments and editor input into interchange. Clients can either submit to a local repository or directly submit to a remote (and potentially central) repository.

Why git?

Sat, 26 May 2007 21:51:02 GMT

I tend to jump from project to project a lot, so it's nice for me to be able to keep track of that by being able to see the separate changes I've made over time as well as a description of what I changed. So rather then keep around a set of patches and a log book, I use a version control system. My choice of vcs/scm/rcs is git. Git is pretty cool, it's distributed, very fast written in C and sh with minimal external dependencies, and it's insanely powerful.

Git provides a version control system that's implemented with the UNIX philosophy in mind. Instead of providing a monolithic tool that does a set of operations that make a vcs possible, git provides all the tools that make up the monolithic tool. This makes it very easy to string together git tools to do nearly anything I could want. An example of this is a tool like git-rev-parse. git-rev-parse returns a commit hash given a name or other identifier. For example, if I want to know what commit hash is the parent of the HEAD of my repository I can do something like:

> git-rev-parse HEAD^
738865616f288809a46bc9b18ec0a90b649892d2

In and of itself, git-rev-parse is useful but does not do much for me, but most of the git tools which accept a commit hash use it to parse the commit hash, so I can pass HEAD^ to git-log and I will get a log of changes to the given repository starting at the commit which is the parent of the head of my repository. This makes writing any kind of tool around git very easy since the actual mechanism is laid bare for manipulation and can be tweaked appropriately without much hassle.

I started using git because xmms2 switched from BitKeeper to git and so I had to learn git to be able to work on xmms2. At first I was a little put off by the low-levelness of it, but two things happened which have generally changed by mind about this:

the wrappers (or porcelain in git speak) around the low-level tools (or plumbing) became a lot better over time. When I started using git, [cogito]i(http://git.or.cz/cogito/) was basically necessary unless you were prepared to really sit down and get to know the git tools to be able to get anything done, but by now I don't even have cogito installed anymore and find that the git tools provided can do near anything I need in a way that I can actually grok.
I started to understand git a lot better. This came from playing with the git tools and realizing how a lot of the stuff worked. While I can't entirely explain this insight I can provide some tips and point out some good resources. The first thing to understand is that git doesn't really track files per-se but rather content. Which means, if you move a file, git knows you moved it because the content changed filename not because the filename changed. This is explained in a lot more detail in a post by Linus on the git mailing list. The other tip I can provide is to be aware that git follows references for you a lot. Just like git-rev-parse will figure out hashes for you, other git tools will figure stuff out for you to, so if you pass git-ls-tree a commit hash, it will read the tree hash out of the commit hash and follow that for you. This makes playing with the git tools really easy because you don't spend all your time chasing the correct hashes around.

So part of my purpose in writing this was to provide some reasons to the FreeBSD folks why git might be a good choice for a replacement for CVS. These views come from my view as a ports committer, but I think that at least some of them will apply to the src folks as well. While the ports tree is usually just hammered with a bunch of small commits all day, there are often those very large commits that can take a while to happen and cause a big fiasco. Often times we lock ports during these events to keep from introducing conflicts. This would be a great place to use branches. these kind of earth shattering changes could be worked on in a branch and then merged into the main repository this way he history of the development of the big change could be maintained in our repository instead of disappearing in one big commit. Git also provides a lot of helpful tools for easing merging which would make bring these big changes in even easier.

Distributed revision control also provides some great advantages during a freeze period. Work that is not appropriate during a freeze can be done locally in a branch and then merged into the mainline when the freeze ends. Commits that are appropriate during the freeze can be committed onto the mainline or into a branch awaiting permission to push them to the master repository. As it stands, committers currently request permission during a freeze from portmgr to commit to ports by submitting a patch and waiting for a reply. Git provides a number of tools specifically designed for submitting patches for approval. It'd also be possible for portmgr to pull commits in from committers during a freeze period.

The xorg 7.2 update was done by doing the bulk of the work on a git repository which was later merged into cvs when the work was done. This made it a lot easier to merge changes in and keep the xorg work on an up to date version of ports before D-day came. I'm taking the same approach with the Python 2.5 update, all work is being done on a branch of ports.git with periodic merges from master to keep the work up to date.

Being aware that I know considerably less about the src process I'd still like to point out some things that I think would make the src development process easier. Merges from current are made considerably easier by using git-cherry-pick which makes it possible to pick particular commits to be commited onto a branch. So a merge from current would just be cherry-picking a commit from master to the STABLE branch. Much of the work that's done in perforce could easily be done with topic branches in git as well.

There is a lot of prejudice attached to git for being a) a bit confusing at first, and b) related to Linux that I really implore everyone to look past and mess around with it for a while. Git is incredibly powerful and works really well for many a development process.

Do not mix sync and async when writing an XMMS2 client

Mon, 28 May 2007 13:11:43 GMT

Having worked on XMMS2 for a long time I've seen many client developers come and go, and seen some really amazing clients being written. The one thing that we see again and again in this development process is developers not understanding the way xmmsclient works entirely and being confused as we over and over tell them "don't mix sync and async." So I'm going to explain why one shouldn't mix sync and async and what exactly that means.

XMMS2's client library (xmmsclient) implements an asynchronous binary IPC protocol (which I've talked about in a previous blog post). Essentially being asynchronous means that clients are returned a deferred result whenever they make a request to the XMMS2 daemon. We wrap this deferred result in an xmmsc_result_t object (XMMSResult in the Python bindings). Upon making a request an xmmsc_result_t is initialized and registered with the IPC result register by use of the xmmsc_ipc_result_register method. When the daemon has a response it writes it to the clients IPC transport (tcp or unix sockets at this point). This data is read in by the xmmsc_ipc_io_in_callback which calls xmmsc_ipc_exec_msg with the new IPC message. xmmsc_ipc_exec_msg looks the corresponding result up in the IPC result register and calls xmmsc_result_run with the message which is then parsed and made accessible by various xmmsc_result_get_ methods depending on the resulting type. Most of this is pretty immediate except for the waiting for the response from the daemon and this is where the asynchronous part comes in.

When data is received on the IPC transport xmmsc_ipc_io_in_callback needs to be called. To do this the file descriptor of the IPC transport needs to be monitored and xmmsc_ipc_io_in_callback needs to be called when data is ready to be read. The same goes for xmmsc_ipc_io_out_callback which needs to be called when data is ready to be written on the file descriptor. There are a number of ways to monitor the file descriptor which aren't all that important to understand. In an asynchronous setup an event loop is attached to the file descriptor and calls xmmsc_ipc_io_in_callback when there is data to be read. Usually an event loop will make provisions for monitoring arbitrary file descriptors because event loops should not be mixed since they can confuse each other. Which rolls us right into why sync and async should not be mixed: synchronous IPC in xmmsclient is done with a select loop.

When xmmsc_result_wait is called on an xmmsc_result_t a while loop is setup up which calls xmmsc_ipc_wait_for_event until a parsed result is present in the xmmsc_result_t. Following over to xmmsc_ipc_wait_for_event we run into:

[...]
if (select (fd + 1, &rfdset, &wfdset, NULL, &tmout) == SOCKET_ERROR) {
        return;
}

if (FD_ISSET (fd, &rfdset)) {
        if (!xmmsc_ipc_io_in_callback (ipc)) {
                return;
        }
}
[...]

So, xmmsc_ipc_wait_for_event sets up a mini-event loop to wait for data to read on the IPC transport which is then handled by xmmsc_ipc_io_in_callback. What this ultimately means is that by mixing sync and async the developer is mixing event loops, which can end up interacting poorly with each other and making a big mess of things.

So basically, if you're developing an XMMS2 client you should either use an event loop or use xmmsc_result_wait but you should NOT use both because they are not meant to play nice together and you can not be sure of what they might do to each other.

My rant on the state of the computer interface

Thu, 21 Jun 2007 18:30:24 GMT

WARNING: This is probably going to be a really long, angry, and potentially upsetting rant, involving an issue I have been interested in and done various bits of development on in the past.

A long time ago in a place about 8 hours south of me, some engineers from Apple went to visit an office in Palo Alto and saw three things: Smalltalk, Ethernet, and the Alto. This is a historically significant event in that it drove Apple to notice the first two and kind of play with them in their future products, and to consumerize the third. I think the first mistake was decoupling the first thing, Smalltalk, from the concept of the third, the GUI, and I think this decision has left interface development in a bit of a rut.

I'm going to lead with a bit of theory and then drive into a lot of examples. My theory is going to be based on some assumptions on my part which I imagine others will not agree with:

As a basic concept the UNIX philosophy works very well
One interface for a particular system (IM, Mail, etc) does not suit all tasks using that system.
Almost all uses of the computer interface involve visualization and manipulation of information
Organizing the information I intend to visualize or manipulate by metadata is a better way of organizing data then a single name.

While not exclusively necessary to buy into my argument the following issues probably play a role:

Though people may not want to believe it, computers still have limited resources
User-friendly is too often an excuse to take capabilities away from those that can make use of them
Obligatory configuration is the root of many evils.

With that being said, I'll touch on various things I've seen, worked on , or been involved with. My first target is going to be a project I became involved with in 2005 whose architecture exemplifies what I think is a better way of handling interface development and organization of data. The project specifically is XMMS2. XMMS2 is a media library managed music player that provides a daemon, xmms2d, which various clients can connect to and manipulate. The first right decision made was to abstract functionality into clients. If for example, one wanted to collect album art for ones music collection this functionality would be handled by a client which would write to the medialib instead of by a plugin to the daemon. Currently this functionality is being developed further with the concept of Service Clients. Service clients open up a whole new world of functionality by allowing clients to issue method calls to any client which registers a service and set of methods. This kind of functionality is why we jokingly refer to XMMS2 as the music-playing microkernel. What this means though is that we get right at my second thesis and at my first and second personal views. The way in which I work with XMMS2 is really set around what exactly I'm trying to do. When I want to browse my music collection I would prefer something visual where I could walk through the collection and search it in various ways, but when I just want to pause a song I can either have a simple playback control client or I can use the command line client and simply issue xmms2 pause at any terminal. I can even do it from a python interpreter if I for some reason need to:

import xmmsclient
xc = xmmsclient.XMMSSync()
xc.connect()
xc.playback_pause()

Which leads right into the second big advantage of the open interface design of XMMS2. I can integrate XMMS2 into any other system by nature of its design instead of having to hack some kind of remote interface like so many older applications have or hoping that the developers of an application provide a usable API with dcop, or dbus, or whatever the trendy remote interface standard is today. The API is used by anyone developing a client and so it has a much higher standard of usability then a secondary API tacked on to provide ability to put the current song you're listening to in an email or a blog post.

The client-server design and open common API are not the only advantages that XMMS2 provides. The media library in XMMS2 has two amazing innovations:

Use of a sourced entity-attribute-value model to which any client is allowed to write
The Collections organization and querying system

XMMS2 indexes metadata off of media into a sqlite database whose Media table schema is something like: id, key, value, source. This is basically a Dictionary hash table, or associative array depending on your language of choice. There are a few disadvantages off the top of my head:

Querying this is slow because in SQL it requires a lot of self JOINs on the Media table
Typing of the value is difficult and ominous at best

The advantages are much more. Firstly, any sort of data can be indexed and made available to the user, such that a user has the ability to arbitrary tag their music in their own way. A user might decide to make a list of people_who_like a certain song, and then use this information later to build a playlist for a party based on whose attending. A playlist that they think most of their friends in attendance will be happy with the selection. More importantly it reduces development headaches in that we don't need to provide an upgrade path for the database every time we decide that we want to index a new tag property from the media files, and also so that we don't end up with a massive list of columns.

Onto our second friend, the Collection. Collections provide a way to organize music by certain constraints and then later query for a list of matching songs. More details about the specifics can be read in the article linked above which gives lots of pretty examples of the capabilities that this functionality provides. What's really more important is the basic idea of allowing arbitrarily constrained collections of metadata which can be linked together and saved, and then queried later. The results are thus always up to date when a query occurs.

Having gone into how I think XMMS2 does the Right Thing (tm), I'd now like to discuss some other systems some of which also do the Right Thing (tm), possibly in a different way, and some of which try to do it, but fail by being too unwillingly to break down the barriers that have been built up so strongly in the past.

The basic concept of cross-application integration has been very popular in the recent past. Apple highlighted the fact that Mail.app could tell you if the person to whom you were writing an email was on AIM at the time you were writing that email. The GNOME crowd quickly picked up on this as some kind of killer feature and offered a large bounty to see it implemented between Evolution and gaim. Initially such functionality was done by providing a simple gaim-remote command line tool and library which could be used to control certain specific functions of gaim in a way that facilitated this exact functionality. More directly the plan wasn't really to open up gaim's interface, but rather just to make this one feature possible. It's really an example of missing the big picture of cross-application integration and instead only picking up on already developed use-cases for it. Apple was most likely able to implement this functionality by using OpenStep's Distributed Objects functionality which allows the exporting of objects over some kind of channel so that other applications can manipulate them. (I assume this is how it was done since the functionality was already there and it seems silly to reinvent the wheel in this case.) NeXT had used this same functionality for collaboration tools where one user could watch another developer working transparently over the network. This functionality was used in the development of Doom and Quake.

Mind you, I'm not championing NeXT and certainly not Apple. I think that Apple while having the ability to really open up interfaces and create a integrated overall interface has focused too much on separate abstract applications, and consumer friendliness (eye candy, music, video). This was not always the case though. I consider the Apple Newton to be a champion of the kind of interface for which I'm calling.

What's so special about the Newton? The Newton actually is amazing in two ways:

Integration of objects between the various tools on the Newton
Not so novel ways of managing and visualizing information that seem to be avoided or forgotten

The Newton didn't use files for storing data but rather had an object database which allowed objects to reference each other and store fairly arbitrary data. This made it easy to extend the address book by adding new properties which could be made useful by third party add-ons to the builtin applications. This also allowed objects to reference each other, so an address book entry could be included in a note such that it linked directly to the address book instead of just including the information about the person. This way modifying the information in the note could potentially change it in the address book, and changes in the address book would be propagated back to the note. This is the kind of functionality that should really be focused on when the issue of cross-application integration comes up. More then just killer-features it should, nay, needs to be ubiquitous: something that's trivial for a developer to use, and hopefully accessible enough that a normal user has this functionality available to them with no or minimal programming skills.

The second issue is also rather cool, but nothing special. Similar functionality was presented in Doug Engelbart's Mother of all Demos. In particular the issue being referred to is the structural nature of lists in Newton's Note program. The idea of being able to collapse, reorganize, and re-visualize list items is a really worthwhile feature. Engelbart demonstrates some really cool functionality such as changing a hierarchical list into a flat list, and back again. He even demonstrates visualizing that list as a tree, and jokingly ends the demonstration of the list, which in this case was a shopping list, with a presentation of line drawn map of the path he would have to take home to pickup all the items he has on the list. What was a joke in 1968 (I repeat 1968) is very possible today. By applying an address property to certain list headers it would be fairly easy to provide a Google Map view of the path one would have to take to get home.

Diverging into the Mother of All Demos, it seems clear that many of the ideas presented in this presentation were lost in favor of fawning over the Mouse and video conferencing. Engelbart's focus on dealing with "wicked problems" would have pushed computer development into an entirely different arena where the focus was on tools that were more general and could be put together to deal with specialized problems, instead of focusing on the development of applications to address specialized problems. The tool architecture is something that was heavily utilized in UNIX development but has had little appearance in the world of the graphical user interface. There are a few (partial) exceptions to this: Plan 9 being a classic example, but in general the UNIX philosophy itself has even begun to break down in the development of CLI tools. A classic complaint about git is its toolkit design, which amounts to over 100 separate commands. The basic concept being that a certain set of them (the plumbing) can be strong together to create the actual functionality of git. For example the commit process is actually the combination of git-write-tree which writes the index to a tree object followed by git-commit-tree which writes a commit object to the repository which references the tree object (There is slightly more involved in the actual process done by the git-commit tool, but that is the basic principle.) This tool based approach is seen as a drawback with constant citations of how it's difficult to use and not user friendly. Those that make this claim are missing the point. Scripts (porcelain in git speak) exist to consolidate these processes for the user, but the toolkit approach allows a level of power and flexibility that are impossible if the functionality is internalized into a closed interface.) I'd go further to say that the closed interface makes it harder for the user to actually do what they want to do.

The entire user-friendliness argument amounts to what I consider a general disrespect for the user. It basically says that the user is not smart enough to learn, or that powerful functionality is usable by such a small subset of people that use an application that it amounts to not being worth having. This is why the open interface is a truly superior approach: generic cases can be easily be built on top of the open interface so as to make for something that's generally considered easy to use, but the interface becomes immediately open to anyone that wants to just play with it. I'm going to amazingly cite an example provided by Microsoft that follows along such a line.

Microsoft's Windows PowerShell exports much of the .NET classes to a user accessible shell with a simple scripting language. This is amazing in a lot of ways, specifically it allows a moderately competent user to experiment with the .NET classes and interfaces from a fairly non threatening environment (no need to compile, instant error notification, etc), and thus gives them more power over their computers functionality and capabilities. Some examples of this power presented directly to the user are:

PS> $rssUrl = "http://blogs.msdn.com/powershell/rss.aspx"
PS> $blog = [xml](new-object System.Net.WebClient).DownloadString($rssUrl)
PS> $blog.rss.channel.item | select title -first 8

Here access to the .NET objects are used directly to download a string and then cast it as an XML object which is then queried with a simple syntax for querying. This is functionality that an eager user could learn to do, or that could easily be scripted, providing a function that provides a title and URL for the top eight blog posts on a RSS feed provided by the user. This could even be pushed into a graphical application which presented the information as a list view. Immediately power is made accessible to a user that is willing to try and learn, with access to millions of examples all over the Internet which they can adapt to better suit their needs (I learned to program by a similar process). If interfaces are locked up and hard to work with and learn then we end up with people less likely to try and experiment with them and we get the kind of people that we always fast "Joe Blow" user as. If you assume people are stupid, you don't give them a chance to change that.

This has mainly been about my first issue, that toolkit based design is more adaptable and provides more power directly to the user and also allows for better integration of functionality. I think that by approaching software development from the first issue, the others can in some ways fall into place but I would like to address them more directly.

I will address my second issue by giving a use-case situation that I hope will point out a common issue. I'm looking at a website for example Portland, Oregon on Wikipedia, and I see this table of temperature and rainfall data. I personally would rather see a graph of this data, as I visualize this sort of thing better graphically. I'm presented with a few options. If I'm on windows I might cut and paste the data into excel and tell it to spit out a graph, on *nix I would load up gnuplot and input the data. It's clear that the ability to visualize this information in different ways is desirable and while I have viable options to deal with this problem, both of them are time consuming (one more so then the other), and require that I make decisions about the data to be graphed. HTML is a semantic format, the table is actually marked as a table and so the ability to convert a table of numbers into a graph (with some best guesses made about how to group the data and the ability to be more specific if necessary) could easily be implemented with open access to the table DOM object. In general these issues arise often, with various methods of presentation being better suited for different people and different situations and too often we focus on unified presentation instead of specialized presentation. As discussed before lists are another common structured format that would be suited to various forms of visualization depending on person and situation.

The computer is an information manipulator. It's designed to organize and access information. It should be thought of that way. Playing music on a computer is an example of this, much more so then a stereo is a information manipulator. Most media library music playing tools are focused on presenting the organization of your media instead of the actual playing of that media. Playlists exist to better organize the music, cover art exists just as much to ease recognition of music as it does to provide eye candy. These capabilities are often ignored or missed. The simple office mailbox has none of the sorting or organizational capabilities provided by even some of the most basic mail programs. With that in mind, those organizational capabilities should be a focus of computers. With the emphasis on relations between information and on providing information and grouping that is better suited to the person whose information is being organized. This is paramount, just because it works for me, doesn't mean it will work for you, so the focus should be on providing a system so dynamic that it can be adapted for whomever is opting to use it.

Filenames don't cut it, and embedded metadata is barely (and often not) enough. Filenames require you to essentialize all of the properties you associate with a piece of information into a fixed string. While possible to serialize various properties into the filename it makes for something difficult to query and to identify ultimately. Instead it makes more sense to catalog the metadata for various bits of information and allow it to be queried in various ways. This capability should necessarily be integrated wherever possible so that when looking at an email you can quickly jump to other emails by that sender or other information with a relation to the sender. Capabilities should exist much like XMMS2's collections to store constraints to be used later, or build them on the fly, and all of the various operators should fit together so that an extra level of filtering can be applied on top of the current one. The current searching systems I've seen (spotlight, beagle) don't really go far enough, and their functionality is rarely available outside of the specific applications. Apple has been making strides to shoehorn (and I use that term with a heavy hand) spotlight into various other parts of the OS, but nothing that's exactly amazing, and in general I think spotlight is too oriented around searching by file and a subset of available file metadata.

I probably upset someone. Sorry about that, maybe it's time to consider that what was revolutionary twenty years ago is tired by now, and that the foundation on which work is done itself is flawed. I had initially intended this article to be a call for ideas, help, and support on some stuff I have sitting around, but I decided that it would be better to present a manifesto of just what is wrong, and what possibilities there are to fix it. I want this manifesto to be thought of as a working draft of the possibilities for the future instead of a rock solid statement on the way things should be done. I will post about some of the projects I am working on that follow this general philosophy later.

A service oriented user interface

Sat, 23 Jun 2007 08:36:13 GMT

In my last post, I kind of dumped a lot of ideas that aren't necessarily as obviously related as I had initially thought. With that in mind, I'm going to have seperate posts to describe many of the issuses in the hope that the relation will become evident.

I'm going to start with what I think is the most important issue behind a revolutionary change in development enviornment. That is specifically the idea of replacing applications with services that provide the essential functions of the applications in a way that the user is able to build an interface around them. Again, my example is XMMS2, where playback and media managment are handled by a service for which various interfaces (or clients) can be made. This means that interfaces can be made for the exact task at hand (I don't need a media library browser to pause my music.) This concept is exactly the direction I think development of the computer interface should go in because it assumes from the begining the basic thesis that, users have personal preferences, and certain interfaces are better suited to certain tasks. This means that much more lightweight interfaces can be used generally, with the ability to pull out something more heavy weight if it's required. It also means that if you have a bunch of services, by their very architecture they can interact with each other.

The service-oriented interface would be unfamilar. The ability to build interfaces for each task would be a common process, indeed the interface building interface, which would allows for connecting various services and visual or textual components together, would be the center point of the user experience. Being fully aware that many tasks are going to require similar interfaces, the ability to save prebuilt interfaces and to draw from a library of already implemented interfaces would be trivial. While the interface building interface would be central, a necessary feature of each service would be a relativly accessible interface for most level of programmer. Direct programming of interfaces should not be hidden nor required, and should provide a welcoming enviornment to those who wish to learn about building interfaces without the interface building interface. Access to the services should be available to as many languages as possible so that those that can already program will be able to comfortably work with the services. It's a mistake to pigeonhole the services to one toolkit, framework, or language. That defeats the purpose of the service-design which is to try as much as possible to allow user's personal preferences to be possible.

While the interface building interface is a bit of a pipe-dream, services are not. Their design makes it possible to use them in nearly any situation. With that in mind, I've begun to write a mail service (which currently indexes headers from my mailboxes.) The mail service will ultimately provide the ability to send mail, as well as retrieve mail from remote and local mailboxes indexing headers from that mail and providing the ability to search, organize and read mails as well as be notified of changes to mailboxes. Basically all the functionality needed for an MUA. The mail service will be the first (or second, if xmms2 is included) of many services that will make a new computer interface possible.

Hopefully decopuling services as a concept from metadata and organization as a concept will make it more clear what exactly I'm trying to achieve. I'd really like to spurn further discussion on this topic so, use comments, email me directly, start writing your own services, help me out on the mail service, get involved with xmms2, or tell me I'm insane.

OSCON

Sat, 28 Jul 2007 00:46:05 GMT

I had the chance to spend Wednesday and Thursday at OSCON and I got to see a lot of really exciting stuff. I was happy to see a BSD booth sponsered by iXsystems, where I had the oppurtunity to meet a lot of great folks in the BSD community including Matt Olander from ixSystems, Dru Lavigne, whose blog provides a lot of great advice for those new and old to FreeBSD, Dan Langille who not only puts on BSDCan but also provides us with FreshPorts and FreshSource, and Kris Moore who has done a lot of work to make an awesome desktop OS ontop of FreeBSD 6.2 known as PCBSD. Needless to say, the chance to meet these folks was quite inspiring. Matt and I discussed at length on a number of topics related to marketing FreeBSD for both servers, and PCBSD as a desktop option. iXsystems has actually brought Kris onboard to work full time on PCBSD and while I don't really want that kind of desktop OS, it's really great to see some competition in the BSD world for systems like Ubuntu, and even more great to see some of the amazing stuff Kris has done with this system. A point that Matt made again and again is that companies that use FreeBSD internally tend to be less vocal about it than companies that use Linux, so to anyone who uses FreeBSD at work: let the world know! (We use FreeBSD for servers and desktops at work.)

My entire trip to OSCON wasn't spent at the BSD booth, I had an oppurtunity to see and support a number of other great projects, as well as some that aren't so great. One project that I saw but haven't had a chance to look into in more detail yet was Apatar which seems to be an "industrial strength" Yahoo! Pipes type tool. While mainly focused on being used internally for generating data transformations, I think this kind of tool can provide a great boon to a standard desktop if the scope is readjusted some. The ability to chain small tools together which transform input data is the foundation of the UNIX philosophy and a graphical interface for generating this kind of transformation with not-just textual output is something that could provide a lot of power to any user with a properly designed interface.

I had the oppurtunity to discuss various development issues with a Yahoo! developer, while me demoed a Flex Yahoo Messenger client. While I'm not inspired by anything Flash related, we did have a lively conversation about the powers and limitations of JavaScript and how much ECMAScript 4 sucks. (classes are not something JavaScript needs.)

I had a great discussion with a Pidgin developer (Richard Laager) on subjects ranging from packaging systems, to distributed version control, to GObjectification of libpurple (The feature I want to see the most in Pidgin development.)

The Sourceforge Community Choice awards were quite fun. I was there to represent XMMS2, and while I was disappointed to see us lose, I was quite pleased to see Audacity win, and I had a lively discussion with an Audacity developer about our projects and potential ways to collaborate. though Audacity has a different scope (being for recording and editing) then XMMS2, there are still many great ideas that can be shared between any project working with audio. So congratulations to the Audacity Project for both: producing a great project, and winning the Multimedia Category at the Comunity Choice Awards. (Also, Guido laughed at my joke [don't ask what it was I can't explain the context well enough for it to be funny.].)

Which brings me on to another topic: in a past blog entry, Tobias, one of XMMS2's lead developers, had discussed the idea of having an Open Audio summit, where a bunch of projects involved with open source audio development could get together and discuss topics of interest to music playing projects. Both the EFF and Creative Commons might be interested in sponsering such an event, and it would provide a great oppurtunity to foster a community between these various projects (as well as a place to discuss the MPRIS project.)

Python 2.5.1 now default on FreeBSD

Mon, 30 Jul 2007 02:45:17 GMT

This is just a quick post to note that we have made Python 2.5.1 the default Python version in the FreeBSD ports collection as of about a minute ago.

This change also included a considerable amount of work to make support for Python Eggs more useable in the Ports collection, by provoiding a few macros for working with easy_install registration, as well as support for Python 2.5's egginfo files.

Most, if not all, of the work was done by Hye-Shik Chang (perky@) who has done a great job of getting this change into the tree. I'd also like to thank Pav Lucistnik (pav@) for being really on the ball about getting experimental runs done for us so we could get these changes in with a minimal amount of damage within the ports tree. We know there are still a few stragling ports, and will be working to fix them as quickly as possible. The ones we know of right now are: java-gcj-compat, gsculpt, py25-wxPython-common, gourmet, and scribus.

Please direct questions, requests, and complaints to python@FreeBSD.org

swfdec gets OSS backend

Mon, 06 Aug 2007 15:00:04 GMT

I'd just like to mention that swfdec 0.5.1 was released, and I have taken maintainership of both the swfdec and swfdec-plugin ports. This new release includes two big milestones: the one most important to FreeBSD users is that our very own Eric Anholt has written an OSS backend, so we can now enjoy sound, which is especially important in light of the fact that swfdec can now play embedded youtube videos!

I've tested this and it is works (though I have a severe performance problem with any effects or rendering in firefox which I think is related to my video card). I've heard reports from others that it works as well.

So now the philsophical message: In a way we're lucky as FreeBSD users that Macromedia and now Adobe have ignored providing flash support for FreeBSD, the reason for that is that we are better able to support projects like swfdec that provide a free implementation of these protocols and open them up to everyone not just those fortunate enough to be considered a worthy platform, so even if you have Flash working in Linuxulator, please give swfdec a try and see if it suits your needs, if it does use it instead!

The biggest spam corpus the world has ever seen

Wed, 21 Nov 2007 20:52:14 GMT

It's been a while. My old laptop had a fan fail, and I didn't really have an easy way to recover data off of my old drive, so today I just went through the pain of doing a recovery. Anyway, I now have the data needed to generate my blog, and can thus start writing entries again.

Today, I was thinking about the idea of personalized spam filters. It occurs to me that the advantage of the personalized spam filter is not that you get less spam, but rather that you get more spam but are less likely for your edge case mail to be marked as spam because of its quasi-spamlike contents. I think what's clear here is that spam is always spam. Doctors don't want viagra spam, even if they talk about viagra on email. Financial advisors don't want hot stock tip spam, even if they talk about hot stock tips via email. So, one prevailing attitude is that making it less likely for a user to get his hot stock tip ham marked as hot stick tip spam is a good thing, even if it means he gets more hot stock tip spam. This make sense, except that the best option is to simply not get your ham marked as spam, and the best way to do that is to have a larger corpus.

Generally, spam filters work by using a Bayesian filter which has a corpus of spam and not spam stored in it. In theory the bigger your (valid) corpus the better it is able to match emails as spam or ham. So when everyone has their own training set, your corpus should be less accurate since it doesn't have the volume of the entire userbase. With all that in mind we can present a solution that can be taken in a number of directions.

The basic idea is that everyone makes their corpus available distributedly. This is acheived by running a small daemon which validates a message as spam upon request. A level above this is a central daemon that tracks corpuses. When a request is made to the central daemon a subset of the cluster is sent the message, and their results are aggregated/average/somethinged to come up with a score for that subset of the cluster. Subsets can either be picked randomly from the nodes connected, or a heuristic can be implemented that tends to push certain messages to certain areas of the cluster (so we end up with a financial advisor subnode which deals with messages related to stocks, or somesuch). This basically means that the message is tested across a massive corpus of different types of people which can classify those messages that are spam to all people better than one which might tend to favor an edge case.

To speed up checks (since running a map-reduce operation over a part of the cluster won't be that fast), each message will be stripped of user/host specific information, and a checksum will be calculated, so that if the same spam comes in it need not be tested, but simply marked as spam.