More than one year ago, Danilo Sato and I started a pet project to practice TDD. We ended up not making all that much progress so far, codewise (over one year! So much for "agile"...), but it's been a great learning exercise. It's a Python test-driven implementation of the Stratego game, and Danilo has documented our early programming sessions on his blog (in Portuguese, I'm waiting for him to catch up to start bitching for an English translation, but I'll post stuff about it here as well). We've recently resumed work on it and it should show up both here and at Danilo's.

Anyway, I have since developed a taste for testing (a "teste", you could say) and, when I started cleaning up nnebs, I found myself writing tests for everything as I went through the code (no, it had no tests before; yes, it was my graduation project; anyone can have a Comp. Sci. B.Sc. these days...). Doctests, even, but more on that later.

One thing that struck me as interesting, in this process, came up while I was writing tests for some state-related classes. Nnebs is basically a spam filter, and I had decided its classifiers (as well as a few other classes) should be stateful, that is, they should be able to easily represent a particular configuration and switch back to it when requested. Since a few different classes would be stateful, I created a Stateful class from which classes wishing to be stateful should inherit. So far, so good. All these classes needed to do was inherit from Stateful and define which attributes would make up their states.

To describe a state itself, I created another class (aptly named "State") to hold attributes and their default values. It defined methods to add, delete, set and query attributes. It also defined that two state objects would be equal if their attributes and values were the same.

The State class was about 30 lines of code and worked quite well. However, while writing its tests I noticed they seemed a bit too trivial, which struck me as odd. I was having to write tests like

>>> State(attr1=val1, attr2=val2) == State(attr1=val1, attr2=val2)
True
>>> State(attr1=val1).get('attr1')
val1

That just doesn't seem right :)

I figured I could delegate some of State's behaviour by making it a subclass of "dict". Great, there goes my "get" implementation. And my equality implementation. And my __del__ implementation. And my initialization implementation. And, well, I assume you can take it from here.

So State is now simply a dict. Running the tests confirmed that everything worked as before. My code is 30 lines of bug-food (a.k.a. "code") shorter. And I'm a lot happier with nnebs and more confident in its code.

I started refactoring and doctesting nnebs, today. Last time I coded for it I was in such a hurry, in order to meet my graduation project's deadline, that, well, let's just say I'm not too proud of how it was delivered :P

Anyway, it's nice to play with nnebs once again, and I should put it online again soon.

Ok, third spam comment in a row. Time to integrate coreblog2 with akismet (and maybe later with nnebs). Should be up soon.

Yipes, I changed the workflow for COREBlog2 and failed to notice that most of my previous posts went into a private state. Publishing them resulted in a weird ordering of posts. I still need to fine-tune coreblog's workflow, so Zarquon knows what'll happen then.

Anyway, not to waste a perfectly fine post ranting randomly about COREBlog, allow me to rant structuredly. This is a list of what I'd like to see in coreblog (typing caps gets old fast)  in the near future (i.e., I consider them to be bugs of some sort). I'll try to add these myself and submit them to the coreblog people.

• Proper workflows. Other people have come up with solutions to this problem, I'll study that a bit and try to find a flexible way to accomodate both single and multiple poster blogs. This should include optional moderation of comments and trackbacks.
  
• Hooks for spam handling (such as akismet, but allowing for arbitrary tools - hint, hint) in comments and trackbacks.
  
• Fix coreblog's annoying habit of sending trackbacks before the post is published, and every time the post is saved. BTW, does anyone know what the policy is for sending trackbacks (in the Real World)? Once, when the post is published? Everytime something changes in a published post? I'd favour the former.
• Adding categories while editing the post. Sometimes I tackle a new subject and only after writing the post do I realize I need a new category or two. I should be able to add these categories from the post itself, it's annoying having to save the post, then create each category individually, then go back to the post to add them. A simple textarea, one category id per line, should do the job nicely.

This is what comes to mind, at the moment. I'm sure more'll surface.

What would you like to fix in coreblog? I might just be in the mood to throw it in as well :)

Update: I just realized I'm not using the latest version of coreblog (I'm using 0.9b and there's a 0.982b since April 13). I'll upgrade and see if something on this list was addressed.

I've been refactoring nnebs for a few days, now. Bugs found, some code rewritten, some mercilessly thrown away. But, more importantly, I've been adding doctests as I go.

This is my first real-world go at doctests. One thing I thought I'd miss were fixtures, but I found out I don't (not much, anyway). "Setting up" usually means creating an object, sometimes a few auxiliary variables, rarely anything more complex than that. "Tearing down" nearly always means, well, closing the docstring :). I'd use fixtures if I had them, but its absence doesn't bother me at this point.

I am a bit overwhelmed by the doctests-to-code ratio, though. Two-line methods often get a 10-line doctest. I am being very thorough and I know some of my tests could be more compact, but simulating all usage possibilities helps me rethink the code (specially after not touching it for almost a year). I get a clear view of how the code should work, both in words and in execution. On the other hand, when reading back the now-tested code I feel all those docstrings tend to clutter up a bit.

A silly, over-simplified, quasi-fictional example, but similiar to situations I've been facing these days. Say I have an object representing a word from an email message (remember, nnebs is a spam filter). This object stores the word itself and the number of times this word showed up in spam and nonspam messages. Now I want to define the addition of two such objects as a third object representing the same word, with added occurrence counts.

def __add__(self, other):
    """Returns self + other.
    This sum is a Word object representing the same value
    as self and other, but containing the sum of the 
    occurrences of both.

    >>> w1 = Word('semprini', spam=2, nonspam=5)
    >>> w2 = Word('semprini', spam=1, nonspam=10)
    >>> w1 + w2 == Word('semprini', spam=3, nonspam=15)
    True

    Words with different values cannot be added:

    >>> Word('semprini') + Word('dinsdale')
    Traceback (most recent call last):
    ...
    ValueError: Cannot sum distinct Words

    """
    if self.value != other.value:
        raise ValueError('Cannot sum distinct Words')
    return Word(self.value,
                spam=self.spam+other.spam,
                nonspam=self.nonspam+other.nonspam)

I like this docstring, it's very explicit about what the operation should (and should not) do. But it's also three times bigger than the code itself (more, if you notice I wrapped the last line to better fit this blog's layout), and after half a dozen of these there's basically only docstrings on my screen.

So they will almost surely be moved to a separate file. The doctest module integrates nicely with the unittest one, so a next step might be to follow Leo Rochael's suggestion and mix both. I considered starting the doctests file right away, but I find it helpful to write the examples while looking at the code (remember, this is not test-driven, I already had the code and didn't quite remember what all of it did). Of course, I must commit to being honest with myself and not adjusting the tests to the code, but I started this with refactoring in mind anyway, so more often than not I use the code to figure out what I originally expected it to do, write tests and then fix what's broken or unnecessary. It's always trivial to cut and paste all docstrings into a new file later.

But probably the best of both worlds would be to view the doctests as either part of the code or as one single extended interpreter session. That is, docstring folding ("go away, I want to look at the code!") and a docstring-only view ("go away, code!"). I wonder if (how) xemacs can do that...

I've re-ordered my pet-project priorities to 1) fix coreblog; 2) refactor and test zzbot; and 3) continue refactoring, testing and turning nnebs into an actual spam filter (and not a spam filtering testbed). I've also decided to get involved in python (the interpreter) development and have started reviewing patches, let's see how I'll fit this into the aforementioned priorities.

I put coreblog at the top because I've found myself reluctant to post before fixing some of the issues I encountered. I'm upgrading Plone today as a first step to start digging into coreblog, so expect something on that front soon.

On the other hand, I just realized a nice side-effect of being unable to receive trackbacks: no trackback spam!

I still get the email warning, though, so that won't keep me comfortable for too long :)

I just got back from PyConBrasil[3] (there's a post about it in the oven, should be ready soon) and have already received two invitations to speak at different conferences!

True, one of them was directed at several of pycon's speakers and the other had nothing to do with pycon at all. Still, nice to see the wheels spinning :)

As soon as I am confirmed on any of these, I'll post a note here.

As people who saw the banner that adorned the top of the my page (and, now, the bottom of this post) these last few weeks might have guessed, I was at Joinville from Wednesday (August 29) to Sunday (September 2) for PyConBrasil3. The conference was tons of fun and I got to meet some friends of old and new alike.

My original plan was to blog daily updates, but, well... I need to get myself a laptop :(.  Volunteers? :)

So I'll leave procrastination for later and get on with a short summary.

Update: Change of plans. The post was getting too big and it was taking me too long to publish anything, so I'll write one post per day (that I was there, not in real-time).

Update on the update: apparently I forgot to publish this post, so it won't help building much anticipation :P

Wednesday, August 29th 2007

Intermission:

Python won't let me cross the boundaries of the list, so let me make it clear that, even though the conference started on Thursday, I got there one day earlier for the first public assembly of the Brazilian Python Association (and thus I indexed Wednesday at 0 and Sunday (the day after the conference) as -1).

Danilo came to my place in the morning, we took the car fueling, washing and oil changing, and drove to Guarulhos airport to meet Nate Aune. I didn't know him, but it wasn't hard to spot the american-looking guy underneath a palm tree holding a video camera and staring at his MacBook Pro :)

We waited a few more hours for Leo to arrive (he was returning from a month in Argentina) as his plane was late (how come Leo always manages to be late, even when he has absolutely no say over it??), had a quick spot of lunch at the airport and , by the time we finally left to Joinville, Nate, having arrived at 7am and it being 1pm already, knew the airport staff by name and spoke only in the soft monotonous voice of flight announcements. 7 hours, incessant rain, very dense fog and loads of lorries later, we arrived just in time to... Get lost in Joinville! Some more driving around, we asked a few locals and eventually managed to arrive at Sociesc's campus, where the conference would be held - and the Association's assembly had started about one hour earlier.

We arrived after the voting for the board had already taken place, but there was only one candidate group, which was elected by acclamation - and was the one I would have voted for anyway, so no harm done. There was some interesting discussion afterwards that served mainly to gather points to be addressed in the future. We should make sure that they are. I'd particularly like to see some action in one specific shortcoming of the Python community: we really suck at marketing! I'll probably write more about this later, but I think we lack better-looking websites (including python.org), easier, shorter tutorials (I'm sure there are a few, but they should be featured prominently on python.org) and basically more "wow factor" for newcomers (I believe experienced Python programmers are already reasonably well-served).

Anyway, once all the formalities were over, we carried on with what we were all really there for: dissing Java!

And beer, and food. Of course :)

Speaking of PyConBrasil3, I've put up a bunch of pictures on flickr, using Aline's Pro account. Descriptions are in Portuguese, but, if anyone asks for it, I can translate them.

PS: Yes, my pyconbrasil[3] "list" has a pictures attribute. So it's not really a list, but it provides a list-like interface. That's dynamic languages for you! ;)