Ideally It Wouldn't Matter

Haystack

2006-06-27T19:49:00.000-07:00

Haystack is a product for Plone used to do auto-classification of content. I've talked about it a little bit before but never given much of an introduction to it before.

Haystack is built around libots which is available for many platforms. ots is a set of Python bindings to the library and I've made it available on cheeseshop. This means you can ez_install its

easy_install ots

or you can grab it yourself from either:

cheeseshop or from svn

After that you'll want to get Haystack, the product for Plone that will get you started.

Once that is installed you'll have access to a new portlet that will show you the interrelationships between content on the system and a tool that will give you more control over the analysis of content.

Of interest:

haystack_tool.summarize(unicode, asHTML=False, ...)
return either a unicode or html highlited summary of the text you passed in

haystack_tool.topics(unicode, count=5, ...)
return a list of topics extracted from the content

Its pretty simple to things like auto suggest keywords/subjects with this tool and Bling in conjunction. Topic maps and other fun things are pretty simple as well. As it gets more use there are many options to expand on the kind of classification and clustering that are available.

Have Fun

PeeJays

2006-06-23T17:01:00.000-07:00

PJS (pronounced PeeJays) is the Python equivlent to RJS done in Rails. Its still a little wet behind the ears but is already quite useful.

The intention is to have a language native way to get at the Javascript code your client is running. In this case the language is Python and we expose the underlying Prototype primitives, but it would be possible to re-target this whole deal for something like Dojo or MochiKit.

So what does it do? How do you use it? Pretty simple really. Now just to disclaim again, not everything you might want to do works but enough does that I am still going to show this. All the tests and runtime are pure Python with no Zope dependencies, but in the context of these examples I use a handy FSJavascript object which means we have a file in your Plone skin path with a .pjs extension that will render itself as text/javascript. There is an implicit page object available to FSJavascript objects, but if you were doing this by hand if would just be:

page = pjs.JavascriptPage()
...
print page

Ok, now for a real example, or what goes in the ...

There are a number of very standard things we want to do from PJS.

Access an element of the page by its id.

page[id]

Refer to a Javascript variable directly

page.jsvar

and then call methods on those and pass them around.

page['portlet-news'].highlight(delay=1.0)

which might highlight the news portlet for one second when triggered.

We can also do slightly more complex things. For example suppose we have a Tree widget in Javascript and we want to add elements to it that result from doing a query. An AJAX callback to the server might trigger a .pjs file that looks
as follows.



##parameters=of
catalog = context.portal_catalog
ob = catalog(UID=of)[0].getObject()

files = page.files
files.clear()
for title, data in ob.treeFolderContents():
    files.addItem(title, data)

Looks just like a regular Python Script in Zope. Parameters tells Zope what to demand from the caller (in this case the AJAX request) and then runs as normal.
This produces the proper Javascript which is returned to the client. All in all its pretty simple. You'd have to understand that files is a Tree object and has a addItem methodand you didn't have to worry about marshalling arguments, or escaping code in some strange cross language way (if you don't know what I mean count your blessings).

There are unit and doc tests for this stuff and its still evolving. Its fun to play with, give it a try.

Update:
This is checked into Bling which can still be had from

http://svn.plone.org/svn/collective/Bling/trunk/

For your pleasure...

2006-06-14T23:48:00.000-07:00

Its been a long time since I've posted. I pushed out releases of some software in the hopes that it will be useful to people.

skeletor 0.4.0.dev-r23725

chimera 0.4.0

rbtree 0.7

ots-0.4.2-py2.4-linux-i686.egg

Skeletor is the zope products and cluster generator.

Chimera is the image generation and manipulation tool. Reco did some install docs. Which are helpful, there are a lot of dependencies. There is also an EGG on cheeseshop but this doesn't track the system level deps.

OTS is the wrapper around libots (which is now included) and is used to power haystack, the auto text summarizer. There is so much that can be done here beyond what is currently shown.

rbtree is generally useful. If you need a datastructre that offers the best of lists and dicts and allows things like key driven slices and so on in Python, this is for you.

Video

2006-03-03T17:52:00.000-08:00

Jonah recently blogged about a video review and screen cast Plone and a number of other frameworks. I was happy to see how well Plone compared some of the other frameworks in question. The ArchGenXML folks really deserve a big hand for this one. I am hoping that in conjunction with tools like Skeletor will put filesystem development at or beyond what we see with other frameworks. Our runtime is already far more powerful, its time the Rapid Application Development and Deployment became simpler to those not already in the know.

On a related note Whit and I have put a little more time into Skeletor and hope to make an official release soon. Whit has updated a branch using setuputils and its quite nice. I wrote a tiny lxml driven registry that allows simple merging of XML trees and fragment and scoped name look ups throughout the project hierarchy.

This means two things, one projects will be able to include their own Skeletor plugins if they wish to allow more/different things to be generated for projects. The act of creating a Skeletor plugin is pretty simple, the act of registering it and having it merge questions into the generation process should be as well. This means that if your fancy project includes a set of Five adapters and some standard configuration that would bind those adapters to the projects
new types you should be able to include a plugin that will generate this for users of your project.

The namespaced name lookups have been there since the beginning but have been cleaned up thanks to the recent work. This will let projects, plugins and authors persist variables from run to run of Skeletor at the correct scope. For example, you might generate projects with a fixed default license, email contact information, etc. You won't be asked for this information each time you run it, Skeletor can retain this information for you. The goal is to make this tool as powerful and painless as possible.

Repository reordering

2006-02-08T17:50:00.000-08:00

People requested that I move some code from the objectrealms.net svn and darcs repos into the collective. I've been hoping that people would try darcs to get at some of the code in that repo (which we think is quite good) but people haven't and it doesn't look like this changes. So I am going to fold.

Bricolite:: https://svn.plone.org/svn/collective/bricolite/

Skeletor:: https://svn.plone.org/svn/collective/skeletor

Expect Chimera soon, the newer versions of that have Archetypes widgets included that can do things like render textual images in the font and style of your choice directly from data kept in Archetypes objects.

The mdns patch for Zope

2006-02-08T12:51:00.000-08:00

People asked for it. This is about the simplest version of the thing I could come up with but you will need the Avahi and DBUS Python bindings up and running. After that this patch should apply to the Zope 2.8/2.9 releases pretty easily.

Then you just restart Zope and you should see these services on your network.

diff -ur Zope-2.8.5-final/lib/python/ZServer/FTPServer.py Zope-2.8.5-mdns/lib/python/ZServer/FTPServer.py
--- Zope-2.8.5-final/lib/python/ZServer/FTPServer.py    2005-12-18 23:25:56.000000000 -0800
+++ Zope-2.8.5-mdns/lib/python/ZServer/FTPServer.py     2006-02-08 12:46:15.000000000 -0800
@@ -80,6 +80,7 @@
import marshal
import stat
import time
+from utils import mdns_announce


class zope_ftp_channel(ftp_channel):
@@ -640,6 +641,10 @@
                self.hostname,
                self.port
                ))
+        mdns_announce(None,
+                      self.port,
+                      service="_ftp._tcp")
+

def clean_shutdown_control(self,phase,time_in_this_phase):
if phase==2:
diff -ur Zope-2.8.5-final/lib/python/ZServer/HTTPServer.py Zope-2.8.5-mdns/lib/python/ZServer/HTTPServer.py
--- Zope-2.8.5-final/lib/python/ZServer/HTTPServer.py   2005-12-18 23:25:56.000000000 -0800
+++ Zope-2.8.5-mdns/lib/python/ZServer/HTTPServer.py    2006-02-08 12:47:46.000000000 -0800
@@ -58,6 +58,7 @@
from zLOG import LOG, register_subsystem, BLATHER, INFO, WARNING, ERROR
import DebugLogger
from medusa import logger
+from utils import mdns_announce

register_subsystem('ZServer HTTPServer')

@@ -384,6 +385,7 @@
server_protocol = 'HTTP'
channel_class = zhttp_channel
shutup=0
+    service = "_http._tcp"

def __init__ (self, ip, port, resolver=None, logger_object=None):
self.shutup=1
@@ -397,6 +399,9 @@
    self.server_port
    ))

+        self.mdns = mdns_announce(None, port, service = self.service, info=self.SERVER_IDENT)
+
+
def clean_shutdown_control(self,phase,time_in_this_phase):
if phase==2:
    self.log_info('closing %s to new connections' % self.server_protocol)
@@ -422,3 +427,4 @@

class zwebdav_server(zhttp_server):
server_protocol = 'WebDAV'
+    service = "_webdav._tcp"
diff -ur Zope-2.8.5-final/lib/python/ZServer/utils.py Zope-2.8.5-mdns/lib/python/ZServer/utils.py
--- Zope-2.8.5-final/lib/python/ZServer/utils.py        2005-12-18 23:25:56.000000000 -0800
+++ Zope-2.8.5-mdns/lib/python/ZServer/utils.py 2006-02-08 12:48:02.000000000 -0800
@@ -56,3 +56,39 @@
from medusa import logger
# override the service name in logger.syslog_logger
logger.syslog_logger.svc_name='ZServer'
+try:
+    import avahi, dbus, dbus.glib
+    hostname = None
+    def mdns_announce(name, port, service="_http._tcp",
+                      **kwargs):
+        global hostname
+        bus = dbus.SystemBus()
+        server = dbus.Interface(bus.get_object(avahi.DBUS_NAME,
+                                               avahi.DBUS_PATH_SERVER),
+                                avahi.DBUS_INTERFACE_SERVER)
+
+        g = dbus.Interface(bus.get_object(avahi.DBUS_NAME,
+                                          server.EntryGroupNew()),
+                           avahi.DBUS_INTERFACE_ENTRY_GROUP)
+
+
+        if name is None:
+            if hostname is None:
+                hostname = "%s:%s" % (server.GetHostName(), port)
+            name = hostname
+
+        info = ["%s=%s" % (k,v) for k,v in kwargs.items()]
+        g.AddService(avahi.IF_UNSPEC,
+                     avahi.PROTO_UNSPEC, 0,
+                     name,
+                     service,
+                     "", "", # domain, host (let the system figure it out)
+                     dbus.UInt16(port),
+                     info,
+                     )
+        g.Commit()
+        return g
+
+except ImportError:
+    def mdns_announce(server, name, port): pass
+

Bonjour Zope

2006-02-02T00:47:00.000-08:00

So I thought how nice would it be for Zope to announce itself using Zero Configuration technology on the network. On Linux we have Avahi and DBUS which make it easy to do such publications. Happily they both have simple Python bindings as well. With a small patch I was able to get my Zope servers to announce themselves on startup for HTTP, FTP and WebDav. They will automatically show up in tools like Nautilus (on GNOME) or Finder (on Mac OS X).

This has got to be nice in situations like Sprints where bunches of developers are passing links to DHCP generated address around adhoc networks. Heck, its nice just when I can't remember which port I left something on.

Bling!

2005-12-07T11:38:00.000-08:00

Dynamic this and that... Everyone wants it, why doesn't Plone have it? There are alot of people and groups working on the AJAXification of Zope technologies with varying degrees of success. Here are the main trends I see people pointing at:

Don't invent a new Javascript library
Don't force the developer to learn Javascript

I am sure there are others. Right now the flavors of the day are MochiKit and the Prototype family of products. I would say that both of them are great, they both provide OO abstractions around writing JS, making it more fun. MochiKit borrows ideas from Python and Twisted and is quite nice to program with. Prototype, Scripaculous and OpenRico excel in the library of existing visual effects and widgets. They also have great mindshare around them in the form that they are included with and used in Rails.

Don't invent something new, don't code Javascript
I've been experimenting with both in the context of Plone. Here is how it breaks down. Mochikit might be better if you are writing alot of Javascript. This is not however the goal of my experimentation. I like auto generated forms and widgets and I'd rather just make those come alive than. Prototype and family are already used in this capacity in Rails so there is experience in what makes a logical binding between the server side and the client.

If we are not writing Javascript then what do we write? Well, Plone UI people know TAL and a bit of Python so I thought I'd aim for that. Introduce Bling, some helper code than can be used in TAL to spit out the Javascript you need.
I only wrote this yesterday so its a little halfed backed but here is how it might work. This will create a "checkedField", something that validates server-side on change using AJAX and then writes to or clears an error div on the client.


<label for="foo">FOO</label> <input type="text" id="foo"
 name="foo"\/>
<div tal:replace="structure python:ajax.checkedField(
      here.portal_url() + '/dval', 'foo', error='error')"/>

which might generate something like



<label for="foo">FOO</label> <input id="foo" name="foo" type="text">
<script type="text/javascript">
  new Form.Element.EventObserver($('foo'),
function(element, value) { new Ajax.Updater({success:
                                          $('error'),failure:
                                          $('error')},
                                          'http://site/bling/dval', {
onFailure:function(request){
      new Effect.Appear($('error'),{}); Fie\ld.activate('foo');},
parameters:Form.Element.serialize($('foo')),
onSuccess:function(request){new Effect.DropOut($('error'),{})},
asynchronous:true,insertion:null,method:'post\'} )  })

Fun Prototype callouts you didn't have to write or think about. Most of the kinds of tags you'd expect in rails are here already and I've added a few more to show how something like Plone could carry a standard array of reuable behaviors and visual effects.

Next we change something like Archetypes that already has per-field validation logic and vocabulary support and allow for things like auto-completion and validition from the widget layer with out change.

To show how simple this is I've written it into a product already. This will install the proper scripts into the resource registries in Plone. Just create a new site, install this product and play with the test.pt template in its skin path. The methods of interest are in Bling/ajax.py

To get the bundle you need darcs:


darcs get http://darcs.objectrealms.net/repos/Bling

or you can just look at the repo at
http://darcs.objectrealms.net/darcsweb.cgi/bling

Events, Frameworks and FUD

2005-12-01T00:07:00.000-08:00

First, a little history. Lets start with the question, "Why doesn't Plone have an event system?" For an event system to really work I would suppose that it needs to be used in all or at least many of the major system in play in a given system. In Plone this would mean that CMFish things would need to fire and consume the same events as the rest of Plone, as the rest of Archetypes, as the events as third party products. In other words an events solution needs to be arrived at from the lowest layers or boil the ocean. Well years into this project none of the lower stack levels have put forward anything usable and nothing has been promoted from application land. Does this mean that no one has tried? Of course not, many have, too many have. In fact in Plone land alone we had 3 different 'event systems' of various size and scope in play for the last few years.

Because no previous effort has won enough development mindshare or devotion to become 'the' event system used in the Plone stack people have by and large been reluctant to use any of them till a winner emerges. We've had a stalemate. To make matters worse the promise of Zope 3 and its clean refinement of the Zope architecture and stack has in many ways halted significant progress in Plone land. We are left making incremental improvements to exiting feature without breaking any new ground. And guess what... That's OK, Zope 3 is better and we have quite a few people waiting for it, experimenting with it, and even a few deploying on it.

Back in Plone land though, no events... oh we can abuse the catalog hooks all we want, those are almost in the right place that we can keep adding things to them to keep extending the system but this is no answer. As a result, it was my impression that the term Event has a certain negative stigma around it, a kind of false promise. For marketing reasons I looked for another term. From a technical perspective events as they are dealt with via the catalog hooks have a particular semantic and a real lack of flexibility and yet still some people see these as events. For technical reasons I sought another term.

If I used the term framework I apologize. That's the flavor of the day word for, "I'd like to be victimized (by the community I am trying to help)" and I should know better than to toss it about, but I don't think I ever said that. The code in question is a pluggable routing class for event payloads and a tool that can manage subscriptions persistently. We recently tried to update it to use Five events for payloads and install a bridge between Z3's 'notify' API and the 'fireEvent' call that makes sense in a Z2 Acquisition hierarchy. That's it.

Its small, its using Z3 concepts and code where we can and we asked for feedback. That offer is still open.

Plotting a course

2005-11-24T12:34:00.000-08:00

Whit and I spent some time talking about Archetypes in the context of Zope 3/Five technology. We spoke about where it stands today, why think its been successful and what to try to carry forward with our new toolbox of slings and arrows... I mean hammers and nails.

The fact that archetypes objects in their create/edit code paths might trigger (re)indexing n time where n is greater that acceptable has been considered low hanging fruit for some of the new technologies. Heck it was low hanging fruit 2 years ago, the solution was never very hard, but requires commitment from the development community that we've not been able to make. We've long thought that events were a simple way do things like indexing. When notified that an object has changed (via an event) trigger an action in the catalog to update the indexes. If your event system can do just a little bit more work this problem and a host of other problems have clearer, more understandable solutions.

We've chosen to replace the term event with the concept of messaging. Messaging lacks some of the historical baggage of events in the Zope/Plone world. No one is asking which messaging system we should be using or how messaging system A compares to B or Zope 3 in general. None of those systems have messaging. Events are the primitive that might enable messaging but messaging implies slightly more than just events.

Events can be though of as callbacks using the subject/observer pattern. Object A says that when object B changes it needs to know and its going to trigger some action. This is typically synchronous and immediately triggers the effect. The advantage of this simple system is that object A never need know that object B care about its change or is going to do anything. A's only responsibility is telling the world that its changed. The world is watching.

What happens when we don't need/want synchronous actions? What if object B has some idea of the applications idea of a transaction and only wants to operate on an object after an entire set of operations have been done? For example, what if we change object A 3 times in one request? We might still only want to (re)index it at the end of the request, not the 3+ times that might happen in the current implementation. How do events help us here?

By themselves events won't help you, you'd need to build a higher level idea around them. The traditional Zope answer would be to put the smarts in the recipient, object B. We have examples of this, queued catalogs and a host of other things that leave the mechanics of queuing and deferring events to the recipient, object B in this case. This solution is less than ideal. It violates DRY (Don't repeat yourself), the handling logic is distributed and replicated in each of the places its needed. The patterns must be reused. I could go on, but thats bad enough.

Enter messaging, This can mean a number of things and rather than outline them all I will just show this one specific example. Object A changes, Object B wants to know. Introduce a message bus or a channel or whatever other word makes ya happy. What this means is that Object C listens for Object A's change and B talks to C now. C, the message bus, introduces ideas in the message handling that do things like manage queuing and collapsing events. Suppose that the catalog subscribes to object C, the ObjectChanged channel. Now when Object A changes object C tells B. B says "thanks, but I don't need to deal with this till later, I don't know when later is, you tell me later calling this method". This is called deferring an event.

Before the end of the transaction the deferred messages will all be delivered back to object B at the same time. Object B can now look at all the messages specific to given objects and decide to merge them into one action or whatever. In the case of cataloging it might figure out the set of catalogs and indexes impacted by all the changes and issue the minimal number of actions to the catalog.

We started a bundle for this, https://svn.plone.org/svn/plone/bundles/messaging dependent on the Eventually messaging system that was written at the last snow sprint but never got any play.

Eventually has some reasonable docs you might want to look at. We are changing Archetypes in this bundle to fire events and adding subscribers for things like that catalog. Think how easy it would be to use the same model to collapse storage events for something like the storage layer. It deals with major sources of pain under a single better abstraction.

Patched Planet

2005-11-21T14:35:00.000-08:00

Planet Plone is using the excellent planetplanet.org software to manage its feeds. However I noticed there was a problem dealing with ATOM feeds such as those emitted from Blogger. I've patched out a version of planet to use a newer feedparser and it seems to have fixed the problem in local testing. Hopefully things will look healthier after the server gets update (thanks wiggy).

Plone's License

2005-11-21T12:14:00.000-08:00

Paul and Sidnei are having a discussion about the GPL license and the reaction of people this. While I have many things I can say about how important it is to ensure the free flow of information I'd point those of you that haven't seen Lawrence Lessig's Free Culture to that excellent presentation.

It to those that imply that Plone going the GPL route is somehow stealing that I would add this. The only possible theft of Free Software comes from placing restrictions on it that remove the rights of others to enjoy, explore and learn from the software the same way you did. The license ensures that at each step of the way its distributed with the same freedoms extending to each recipient.

This is complicated when mixing GPL software with other software, obviously. In the cases in which you'd mix with software that would restrict the rights of its consumers, such as mixing with proprietary software, the GPL is pretty clear. Its in the case of mixing with other Open Source software products that things become less clear. If both products intend for example for the works to be distributed with the source always available but define different notions of what makes an aggregate work covered by the distribution terms of the license things grow increasingly complex.

Rather than detail various cases, I'd say this. Its important that software projects define the intent and understanding with which they use the license, and its far better to ship with known, better understood licenses (even with legally binding addendums and exceptions) than to invent your own. The Plone Foundation has a responsibility, as owner of the Plone IP, to more adequately define our intended usage, derivation model, etc, that we can better protect it and the rights of the software's consumers.

Stop Energy

2005-11-16T00:26:00.000-08:00

Some people know the term Stop Energy, others don't. If you don't you should. Its rampant and education is the key to stopping it. Stop Energy is any non-constructive, unreasoned response to an attempt to accomplish something.

There is a common belief that when presented with a new idea, or even an old one, that a request for comments means criticism. Stop Energy is applied without thought of the bigger picture, its an effort to prevent the realization of an idea.

Rather than recast the entire content of the link above I'd add the following:

Stop Energy is bad behavior. When confronted with a new idea try to think about how it can be made to work, try to think about how you can help. If there is constructive criticism to be realized about an idea it will fall naturally out of such an approach.

Skeletor

2005-11-14T22:59:00.000-08:00

Geoff Davis and myself recently participated in a sprint devoted to packaging Plone products and best practices around product development. There have been a number of good tutorials including the Bricolite one that I gave a while back in Vienna. We decided that a tool that could spit out project skeletons for Plone would help people take advantage of some of the better practices collected in the community.

Our tool generates a "Batteries Included" project skeleton. Its our hope that in the short term this will include:

- automatically generating EGGs

- automatic generation of HTML/PDF documentation

- build in testing support with common interfaces tests out of the box

- built in support for pyflake and some other Python testing/analysis tools.

The name of this mighty productivity aid? Skeletor.

Skeletor will be available from my darcs repo in the near term. For now its still in development but if you have ideas its darcs, Check it out and send me a patch.

Process Foundations

2005-11-14T17:26:00.000-08:00

The Plone Foundation board is recommending that Plone developers start blogging about the good works that the community and Foundation Membership are up to. This is important. Plone is a powerful, flexible product but isn't getting the same product love as some of the other web development frameworks on the market today.

I find that many in the community compare us to them where them is a collection of frameworks for doing web presentation on top of simple relational databases. Many of these frameworks are cool in their own right and in their own product spaces but Plone is much more than any of them. Plone is a full fledged Content Management System that has competed with some of the very best commercial systems and won. It offers advanced features for workflow management of content, collaborative content development and a suite of other features that set it apart.

Frameworks that pride themselves on leanness and rapid application development offer those things because they give you the ability to manage so much less complexity. The world is complex and the problems people attempt to solve in it doubly so.

I think proper positioning in the technology market will help developers hunting for frameworks and help to bring additional talent into our playground.