The fact that archetypes objects in their create/edit code paths might trigger (re)indexing n time where n is greater that acceptable has been considered low hanging fruit for some of the new technologies. Heck it was low hanging fruit 2 years ago, the solution was never very hard, but requires commitment from the development community that we've not been able to make. We've long thought that events were a simple way do things like indexing. When notified that an object has changed (via an event) trigger an action in the catalog to update the indexes. If your event system can do just a little bit more work this problem and a host of other problems have clearer, more understandable solutions.
We've chosen to replace the term event with the concept of messaging. Messaging lacks some of the historical baggage of events in the Zope/Plone world. No one is asking which messaging system we should be using or how messaging system A compares to B or Zope 3 in general. None of those systems have messaging. Events are the primitive that might enable messaging but messaging implies slightly more than just events.
Events can be though of as callbacks using the subject/observer pattern. Object A says that when object B changes it needs to know and its going to trigger some action. This is typically synchronous and immediately triggers the effect. The advantage of this simple system is that object A never need know that object B care about its change or is going to do anything. A's only responsibility is telling the world that its changed. The world is watching.
What happens when we don't need/want synchronous actions? What if object B has some idea of the applications idea of a transaction and only wants to operate on an object after an entire set of operations have been done? For example, what if we change object A 3 times in one request? We might still only want to (re)index it at the end of the request, not the 3+ times that might happen in the current implementation. How do events help us here?
By themselves events won't help you, you'd need to build a higher level idea around them. The traditional Zope answer would be to put the smarts in the recipient, object B. We have examples of this, queued catalogs and a host of other things that leave the mechanics of queuing and deferring events to the recipient, object B in this case. This solution is less than ideal. It violates DRY (Don't repeat yourself), the handling logic is distributed and replicated in each of the places its needed. The patterns must be reused. I could go on, but thats bad enough.
Enter messaging, This can mean a number of things and rather than outline them all I will just show this one specific example. Object A changes, Object B wants to know. Introduce a message bus or a channel or whatever other word makes ya happy. What this means is that Object C listens for Object A's change and B talks to C now. C, the message bus, introduces ideas in the message handling that do things like manage queuing and collapsing events. Suppose that the catalog subscribes to object C, the ObjectChanged channel. Now when Object A changes object C tells B. B says "thanks, but I don't need to deal with this till later, I don't know when later is, you tell me later calling this method". This is called deferring an event.
Before the end of the transaction the deferred messages will all be delivered back to object B at the same time. Object B can now look at all the messages specific to given objects and decide to merge them into one action or whatever. In the case of cataloging it might figure out the set of catalogs and indexes impacted by all the changes and issue the minimal number of actions to the catalog.
We started a bundle for this, https://svn.plone.org/svn/plone/bundles/messaging dependent on the Eventually messaging system that was written at the last snow sprint but never got any play.
Eventually has some reasonable docs you might want to look at. We are changing Archetypes in this bundle to fire events and adding subscribers for things like that catalog. Think how easy it would be to use the same model to collapse storage events for something like the storage layer. It deals with major sources of pain under a single better abstraction.
5 comments:
+10! This is a long-needed piece of architecture and I will hold you in high regard as heroes for this. Ignore the current stop-energy. We'll never get to the future by fixing bugs and patching the current stuff (which is undoubtedly good and neccesary too, don't get me wrong..) without ever taking the neccesary step back and thinking things over every once in a while. Scotch for Ben and Whit !
I'm trying to see whether the stack this messaging framework fits. Is this to be a Plone/Archetypes framework or will this be a contribution to Zope 3? This will determine in part what forum the discussion on this should take place in.
I'd prefer it if new frameworks like this were built in the context of Zope 3 first and then ported to Five instead of building on Five directly, though of course sometimes that may be the expeditious strategy.
Martijn Faassen said...
I'm trying to see whether the stack this messaging framework fits. Is this to be a Plone/Archetypes framework or will this be a contribution to Zope 3? This will determine in part what forum the discussion on this should take place in.
I'd prefer it if new frameworks like this were built in the context of Zope 3 first and then ported to Five instead of building on Five directly, though of course sometimes that may be the expeditious strategy.
This is based on something I did in pure Python with the intent that would be usable in Zope3. That said the abstraction between message payload and message routing has been bundled to work well in the Zope2 world for now. Because the idea of message routing is distinct from delivery I can do things like easily support a DOM like event model with bubble and capture phases in the object heirarchy. To use this in Zope3 I suspect we'd just mark the message routing part with an interface so we could adapt it to the environment it was running in.
The code itself is pretty simple. I'd welcome any real feedback.
Much in the spirit of Antoine de Saint-Exupery's maybe overrated quote on perfection, the Zope 3 component architecture has several times been simplified by reducing the number of concepts, not creating new ones. When you don't really have an idea of something quite yet, you intend to overcomplicate, so it's important to simplify .
Take object events, for example. You could come up with a complicated system on how to deal with them so that certain event handlers would only be triggered for certain object types. Zope 3 simply has a dispatcher that makes the event system eat its own dogfood. Solutions like that is what we need. Take simple concepts we have already, make them eat their own dogfood to tacke bigger problems.
This so-called "messaging" doesn't need its own framework really. A simple subscriber that saves events away for later reuse (e.g. consolidation of several events into one) in a thread local would suffice. That makes one function and one ZCML snippet for its registration.
So, if I understand you correctly, this piece of software fits potentially best in the Zope 3 stack, but has specific code to deal with Zope 2, so that part of it probably fits best in Five. Some of it
is so generic it might fit as a standalone Python library, but since you're talking about making Plone have this, I assume you want it bundled somewhere inside the Plone stack eventually.
It may of course be that this may need to
be in an extension to Zope 3 for a while, and thus an extension to Five, until it
can migrate into the core if the community finds it valuable there.
I would not like this to disappear somewhere into the Plone part of the stack right away; if it's valuable and it's in Python and generic, I understood that the Plone community wants to encourage people to work in more frameworky areas of the stack instead of adding it to Plone.
So, why not post this on zope3-dev? It may result in valuable feedback for you and it appears to be the best place.
Post a Comment