Events, Frameworks and FUD

First, a little history. Lets start with the question, "Why doesn't Plone have an event system?" For an event system to really work I would suppose that it needs to be used in all or at least many of the major system in play in a given system. In Plone this would mean that CMFish things would need to fire and consume the same events as the rest of Plone, as the rest of Archetypes, as the events as third party products. In other words an events solution needs to be arrived at from the lowest layers or boil the ocean. Well years into this project none of the lower stack levels have put forward anything usable and nothing has been promoted from application land. Does this mean that no one has tried? Of course not, many have, too many have. In fact in Plone land alone we had 3 different 'event systems' of various size and scope in play for the last few years.

Because no previous effort has won enough development mindshare or devotion to become 'the' event system used in the Plone stack people have by and large been reluctant to use any of them till a winner emerges. We've had a stalemate. To make matters worse the promise of Zope 3 and its clean refinement of the Zope architecture and stack has in many ways halted significant progress in Plone land. We are left making incremental improvements to exiting feature without breaking any new ground. And guess what... That's OK, Zope 3 is better and we have quite a few people waiting for it, experimenting with it, and even a few deploying on it.

Back in Plone land though, no events... oh we can abuse the catalog hooks all we want, those are almost in the right place that we can keep adding things to them to keep extending the system but this is no answer. As a result, it was my impression that the term Event has a certain negative stigma around it, a kind of false promise. For marketing reasons I looked for another term. From a technical perspective events as they are dealt with via the catalog hooks have a particular semantic and a real lack of flexibility and yet still some people see these as events. For technical reasons I sought another term.

If I used the term framework I apologize. That's the flavor of the day word for, "I'd like to be victimized (by the community I am trying to help)" and I should know better than to toss it about, but I don't think I ever said that. The code in question is a pluggable routing class for event payloads and a tool that can manage subscriptions persistently. We recently tried to update it to use Five events for payloads and install a bridge between Z3's 'notify' API and the 'fireEvent' call that makes sense in a Z2 Acquisition hierarchy. That's it.

Its small, its using Z3 concepts and code where we can and we asked for feedback. That offer is still open.

Plotting a course

Whit and I spent some time talking about Archetypes in the context of Zope 3/Five technology. We spoke about where it stands today, why think its been successful and what to try to carry forward with our new toolbox of slings and arrows... I mean hammers and nails.

The fact that archetypes objects in their create/edit code paths might trigger (re)indexing n time where n is greater that acceptable has been considered low hanging fruit for some of the new technologies. Heck it was low hanging fruit 2 years ago, the solution was never very hard, but requires commitment from the development community that we've not been able to make. We've long thought that events were a simple way do things like indexing. When notified that an object has changed (via an event) trigger an action in the catalog to update the indexes. If your event system can do just a little bit more work this problem and a host of other problems have clearer, more understandable solutions.

We've chosen to replace the term event with the concept of messaging. Messaging lacks some of the historical baggage of events in the Zope/Plone world. No one is asking which messaging system we should be using or how messaging system A compares to B or Zope 3 in general. None of those systems have messaging. Events are the primitive that might enable messaging but messaging implies slightly more than just events.

Events can be though of as callbacks using the subject/observer pattern. Object A says that when object B changes it needs to know and its going to trigger some action. This is typically synchronous and immediately triggers the effect. The advantage of this simple system is that object A never need know that object B care about its change or is going to do anything. A's only responsibility is telling the world that its changed. The world is watching.

What happens when we don't need/want synchronous actions? What if object B has some idea of the applications idea of a transaction and only wants to operate on an object after an entire set of operations have been done? For example, what if we change object A 3 times in one request? We might still only want to (re)index it at the end of the request, not the 3+ times that might happen in the current implementation. How do events help us here?

By themselves events won't help you, you'd need to build a higher level idea around them. The traditional Zope answer would be to put the smarts in the recipient, object B. We have examples of this, queued catalogs and a host of other things that leave the mechanics of queuing and deferring events to the recipient, object B in this case. This solution is less than ideal. It violates DRY (Don't repeat yourself), the handling logic is distributed and replicated in each of the places its needed. The patterns must be reused. I could go on, but thats bad enough.

Enter messaging, This can mean a number of things and rather than outline them all I will just show this one specific example. Object A changes, Object B wants to know. Introduce a message bus or a channel or whatever other word makes ya happy. What this means is that Object C listens for Object A's change and B talks to C now. C, the message bus, introduces ideas in the message handling that do things like manage queuing and collapsing events. Suppose that the catalog subscribes to object C, the ObjectChanged channel. Now when Object A changes object C tells B. B says "thanks, but I don't need to deal with this till later, I don't know when later is, you tell me later calling this method". This is called deferring an event.

Before the end of the transaction the deferred messages will all be delivered back to object B at the same time. Object B can now look at all the messages specific to given objects and decide to merge them into one action or whatever. In the case of cataloging it might figure out the set of catalogs and indexes impacted by all the changes and issue the minimal number of actions to the catalog.

We started a bundle for this, https://svn.plone.org/svn/plone/bundles/messaging dependent on the Eventually messaging system that was written at the last snow sprint but never got any play.

Eventually has some reasonable docs you might want to look at. We are changing Archetypes in this bundle to fire events and adding subscribers for things like that catalog. Think how easy it would be to use the same model to collapse storage events for something like the storage layer. It deals with major sources of pain under a single better abstraction.

Patched Planet

Planet Plone is using the excellent planetplanet.org software to manage its feeds. However I noticed there was a problem dealing with ATOM feeds such as those emitted from Blogger. I've patched out a version of planet to use a newer feedparser and it seems to have fixed the problem in local testing. Hopefully things will look healthier after the server gets update (thanks wiggy).

Plone's License

Paul and Sidnei are having a discussion about the GPL license and the reaction of people this. While I have many things I can say about how important it is to ensure the free flow of information I'd point those of you that haven't seen Lawrence Lessig's Free Culture to that excellent presentation.

It to those that imply that Plone going the GPL route is somehow stealing that I would add this. The only possible theft of Free Software comes from placing restrictions on it that remove the rights of others to enjoy, explore and learn from the software the same way you did. The license ensures that at each step of the way its distributed with the same freedoms extending to each recipient.

This is complicated when mixing GPL software with other software, obviously. In the cases in which you'd mix with software that would restrict the rights of its consumers, such as mixing with proprietary software, the GPL is pretty clear. Its in the case of mixing with other Open Source software products that things become less clear. If both products intend for example for the works to be distributed with the source always available but define different notions of what makes an aggregate work covered by the distribution terms of the license things grow increasingly complex.

Rather than detail various cases, I'd say this. Its important that software projects define the intent and understanding with which they use the license, and its far better to ship with known, better understood licenses (even with legally binding addendums and exceptions) than to invent your own. The Plone Foundation has a responsibility, as owner of the Plone IP, to more adequately define our intended usage, derivation model, etc, that we can better protect it and the rights of the software's consumers.

Stop Energy

Some people know the term Stop Energy, others don't. If you don't you should. Its rampant and education is the key to stopping it. Stop Energy is any non-constructive, unreasoned response to an attempt to accomplish something.

There is a common belief that when presented with a new idea, or even an old one, that a request for comments means criticism. Stop Energy is applied without thought of the bigger picture, its an effort to prevent the realization of an idea.

Rather than recast the entire content of the link above I'd add the following:

Stop Energy is bad behavior. When confronted with a new idea try to think about how it can be made to work, try to think about how you can help. If there is constructive criticism to be realized about an idea it will fall naturally out of such an approach.

Skeletor

Geoff Davis and myself recently participated in a sprint devoted to packaging Plone products and best practices around product development. There have been a number of good tutorials including the Bricolite one that I gave a while back in Vienna. We decided that a tool that could spit out project skeletons for Plone would help people take advantage of some of the better practices collected in the community.

Our tool generates a "Batteries Included" project skeleton. Its our hope that in the short term this will include:

- automatically generating EGGs

- automatic generation of HTML/PDF documentation

- build in testing support with common interfaces tests out of the box

- built in support for pyflake and some other Python testing/analysis tools.

The name of this mighty productivity aid? Skeletor.

Skeletor will be available from my darcs repo in the near term. For now its still in development but if you have ideas its darcs, Check it out and send me a patch.

Process Foundations

The Plone Foundation board is recommending that Plone developers start blogging about the good works that the community and Foundation Membership are up to. This is important. Plone is a powerful, flexible product but isn't getting the same product love as some of the other web development frameworks on the market today.

I find that many in the community compare us to them where them is a collection of frameworks for doing web presentation on top of simple relational databases. Many of these frameworks are cool in their own right and in their own product spaces but Plone is much more than any of them. Plone is a full fledged Content Management System that has competed with some of the very best commercial systems and won. It offers advanced features for workflow management of content, collaborative content development and a suite of other features that set it apart.

Frameworks that pride themselves on leanness and rapid application development offer those things because they give you the ability to manage so much less complexity. The world is complex and the problems people attempt to solve in it doubly so.

I think proper positioning in the technology market will help developers hunting for frameworks and help to bring additional talent into our playground.