Haystack

Haystack is a product for Plone used to do auto-classification of content. I've talked about it a little bit before but never given much of an introduction to it before.

Haystack is built around libots which is available for many platforms. ots is a set of Python bindings to the library and I've made it available on cheeseshop. This means you can ez_install its

easy_install ots

or you can grab it yourself from either:

cheeseshop or from svn

After that you'll want to get Haystack, the product for Plone that will get you started.

Once that is installed you'll have access to a new portlet that will show you the interrelationships between content on the system and a tool that will give you more control over the analysis of content.

Of interest:

haystack_tool.summarize(unicode, asHTML=False, ...)
return either a unicode or html highlited summary of the text you passed in

haystack_tool.topics(unicode, count=5, ...)
return a list of topics extracted from the content

Its pretty simple to things like auto suggest keywords/subjects with this tool and Bling in conjunction. Topic maps and other fun things are pretty simple as well. As it gets more use there are many options to expand on the kind of classification and clustering that are available.

Have Fun

PeeJays

PJS (pronounced PeeJays) is the Python equivlent to RJS done in Rails. Its still a little wet behind the ears but is already quite useful.

The intention is to have a language native way to get at the Javascript code your client is running. In this case the language is Python and we expose the underlying Prototype primitives, but it would be possible to re-target this whole deal for something like Dojo or MochiKit.

So what does it do? How do you use it? Pretty simple really. Now just to disclaim again, not everything you might want to do works but enough does that I am still going to show this. All the tests and runtime are pure Python with no Zope dependencies, but in the context of these examples I use a handy FSJavascript object which means we have a file in your Plone skin path with a .pjs extension that will render itself as text/javascript. There is an implicit page object available to FSJavascript objects, but if you were doing this by hand if would just be:

page = pjs.JavascriptPage()
...
print page

Ok, now for a real example, or what goes in the ...



There are a number of very standard things we want to do from PJS.

Access an element of the page by its id.

page[id]

Refer to a Javascript variable directly

page.jsvar

and then call methods on those and pass them around.

page['portlet-news'].highlight(delay=1.0)

which might highlight the news portlet for one second when triggered.

We can also do slightly more complex things. For example suppose we have a Tree widget in Javascript and we want to add elements to it that result from doing a query. An AJAX callback to the server might trigger a .pjs file that looks
as follows.



##parameters=of
catalog = context.portal_catalog
ob = catalog(UID=of)[0].getObject()

files = page.files
files.clear()
for title, data in ob.treeFolderContents():
files.addItem(title, data)



Looks just like a regular Python Script in Zope. Parameters tells Zope what to demand from the caller (in this case the AJAX request) and then runs as normal.
This produces the proper Javascript which is returned to the client. All in all its pretty simple. You'd have to understand that files is a Tree object and has a addItem methodand you didn't have to worry about marshalling arguments, or escaping code in some strange cross language way (if you don't know what I mean count your blessings).

There are unit and doc tests for this stuff and its still evolving. Its fun to play with, give it a try.

Update:
This is checked into Bling which can still be had from

http://svn.plone.org/svn/collective/Bling/trunk/

For your pleasure...

Its been a long time since I've posted. I pushed out releases of some software in the hopes that it will be useful to people.

skeletor 0.4.0.dev-r23725

chimera 0.4.0

rbtree 0.7

ots-0.4.2-py2.4-linux-i686.egg

Skeletor is the zope products and cluster generator.

Chimera is the image generation and manipulation tool. Reco did some install docs. Which are helpful, there are a lot of dependencies. There is also an EGG on cheeseshop but this doesn't track the system level deps.

OTS is the wrapper around libots (which is now included) and is used to power haystack, the auto text summarizer. There is so much that can be done here beyond what is currently shown.

rbtree is generally useful. If you need a datastructre that offers the best of lists and dicts and allows things like key driven slices and so on in Python, this is for you.

Video

Jonah recently blogged about a video review and screen cast Plone and a number of other frameworks. I was happy to see how well Plone compared some of the other frameworks in question. The ArchGenXML folks really deserve a big hand for this one. I am hoping that in conjunction with tools like Skeletor will put filesystem development at or beyond what we see with other frameworks. Our runtime is already far more powerful, its time the Rapid Application Development and Deployment became simpler to those not already in the know.

On a related note Whit and I have put a little more time into Skeletor and hope to make an official release soon. Whit has updated a branch using setuputils and its quite nice. I wrote a tiny lxml driven registry that allows simple merging of XML trees and fragment and scoped name look ups throughout the project hierarchy.

This means two things, one projects will be able to include their own Skeletor plugins if they wish to allow more/different things to be generated for projects. The act of creating a Skeletor plugin is pretty simple, the act of registering it and having it merge questions into the generation process should be as well. This means that if your fancy project includes a set of Five adapters and some standard configuration that would bind those adapters to the projects
new types you should be able to include a plugin that will generate this for users of your project.

The namespaced name lookups have been there since the beginning but have been cleaned up thanks to the recent work. This will let projects, plugins and authors persist variables from run to run of Skeletor at the correct scope. For example, you might generate projects with a fixed default license, email contact information, etc. You won't be asked for this information each time you run it, Skeletor can retain this information for you. The goal is to make this tool as powerful and painless as possible.

Repository reordering

People requested that I move some code from the objectrealms.net svn and darcs repos into the collective. I've been hoping that people would try darcs to get at some of the code in that repo (which we think is quite good) but people haven't and it doesn't look like this changes. So I am going to fold.

Bricolite:: https://svn.plone.org/svn/collective/bricolite/

Skeletor:: https://svn.plone.org/svn/collective/skeletor

Expect Chimera soon, the newer versions of that have Archetypes widgets included that can do things like render textual images in the font and style of your choice directly from data kept in Archetypes objects.

The mdns patch for Zope

People asked for it. This is about the simplest version of the thing I could come up with but you will need the Avahi and DBUS Python bindings up and running. After that this patch should apply to the Zope 2.8/2.9 releases pretty easily.

Then you just restart Zope and you should see these services on your network.

diff -ur Zope-2.8.5-final/lib/python/ZServer/FTPServer.py Zope-2.8.5-mdns/lib/python/ZServer/FTPServer.py
--- Zope-2.8.5-final/lib/python/ZServer/FTPServer.py 2005-12-18 23:25:56.000000000 -0800
+++ Zope-2.8.5-mdns/lib/python/ZServer/FTPServer.py 2006-02-08 12:46:15.000000000 -0800
@@ -80,6 +80,7 @@
import marshal
import stat
import time
+from utils import mdns_announce


class zope_ftp_channel(ftp_channel):
@@ -640,6 +641,10 @@
self.hostname,
self.port
))
+ mdns_announce(None,
+ self.port,
+ service="_ftp._tcp")
+

def clean_shutdown_control(self,phase,time_in_this_phase):
if phase==2:
diff -ur Zope-2.8.5-final/lib/python/ZServer/HTTPServer.py Zope-2.8.5-mdns/lib/python/ZServer/HTTPServer.py
--- Zope-2.8.5-final/lib/python/ZServer/HTTPServer.py 2005-12-18 23:25:56.000000000 -0800
+++ Zope-2.8.5-mdns/lib/python/ZServer/HTTPServer.py 2006-02-08 12:47:46.000000000 -0800
@@ -58,6 +58,7 @@
from zLOG import LOG, register_subsystem, BLATHER, INFO, WARNING, ERROR
import DebugLogger
from medusa import logger
+from utils import mdns_announce

register_subsystem('ZServer HTTPServer')

@@ -384,6 +385,7 @@
server_protocol = 'HTTP'
channel_class = zhttp_channel
shutup=0
+ service = "_http._tcp"

def __init__ (self, ip, port, resolver=None, logger_object=None):
self.shutup=1
@@ -397,6 +399,9 @@
self.server_port
))

+ self.mdns = mdns_announce(None, port, service = self.service, info=self.SERVER_IDENT)
+
+
def clean_shutdown_control(self,phase,time_in_this_phase):
if phase==2:
self.log_info('closing %s to new connections' % self.server_protocol)
@@ -422,3 +427,4 @@

class zwebdav_server(zhttp_server):
server_protocol = 'WebDAV'
+ service = "_webdav._tcp"
diff -ur Zope-2.8.5-final/lib/python/ZServer/utils.py Zope-2.8.5-mdns/lib/python/ZServer/utils.py
--- Zope-2.8.5-final/lib/python/ZServer/utils.py 2005-12-18 23:25:56.000000000 -0800
+++ Zope-2.8.5-mdns/lib/python/ZServer/utils.py 2006-02-08 12:48:02.000000000 -0800
@@ -56,3 +56,39 @@
from medusa import logger
# override the service name in logger.syslog_logger
logger.syslog_logger.svc_name='ZServer'
+try:
+ import avahi, dbus, dbus.glib
+ hostname = None
+ def mdns_announce(name, port, service="_http._tcp",
+ **kwargs):
+ global hostname
+ bus = dbus.SystemBus()
+ server = dbus.Interface(bus.get_object(avahi.DBUS_NAME,
+ avahi.DBUS_PATH_SERVER),
+ avahi.DBUS_INTERFACE_SERVER)
+
+ g = dbus.Interface(bus.get_object(avahi.DBUS_NAME,
+ server.EntryGroupNew()),
+ avahi.DBUS_INTERFACE_ENTRY_GROUP)
+
+
+ if name is None:
+ if hostname is None:
+ hostname = "%s:%s" % (server.GetHostName(), port)
+ name = hostname
+
+ info = ["%s=%s" % (k,v) for k,v in kwargs.items()]
+ g.AddService(avahi.IF_UNSPEC,
+ avahi.PROTO_UNSPEC, 0,
+ name,
+ service,
+ "", "", # domain, host (let the system figure it out)
+ dbus.UInt16(port),
+ info,
+ )
+ g.Commit()
+ return g
+
+except ImportError:
+ def mdns_announce(server, name, port): pass
+

Bonjour Zope

So I thought how nice would it be for Zope to announce itself using Zero Configuration technology on the network. On Linux we have Avahi and DBUS which make it easy to do such publications. Happily they both have simple Python bindings as well. With a small patch I was able to get my Zope servers to announce themselves on startup for HTTP, FTP and WebDav. They will automatically show up in tools like Nautilus (on GNOME) or Finder (on Mac OS X).

This has got to be nice in situations like Sprints where bunches of developers are passing links to DHCP generated address around adhoc networks. Heck, its nice just when I can't remember which port I left something on.

Bling!

Dynamic this and that... Everyone wants it, why doesn't Plone have it? There are alot of people and groups working on the AJAXification of Zope technologies with varying degrees of success. Here are the main trends I see people pointing at:

• Don't invent a new Javascript library
• Don't force the developer to learn Javascript

I am sure there are others. Right now the flavors of the day are MochiKit and the Prototype family of products. I would say that both of them are great, they both provide OO abstractions around writing JS, making it more fun. MochiKit borrows ideas from Python and Twisted and is quite nice to program with. Prototype, Scripaculous and OpenRico excel in the library of existing visual effects and widgets. They also have great mindshare around them in the form that they are included with and used in Rails.

Don't invent something new, don't code Javascript
I've been experimenting with both in the context of Plone. Here is how it breaks down. Mochikit might be better if you are writing alot of Javascript. This is not however the goal of my experimentation. I like auto generated forms and widgets and I'd rather just make those come alive than. Prototype and family are already used in this capacity in Rails so there is experience in what makes a logical binding between the server side and the client.

If we are not writing Javascript then what do we write? Well, Plone UI people know TAL and a bit of Python so I thought I'd aim for that. Introduce Bling, some helper code than can be used in TAL to spit out the Javascript you need.
I only wrote this yesterday so its a little halfed backed but here is how it might work. This will create a "checkedField", something that validates server-side on change using AJAX and then writes to or clears an error div on the client.

<label for="foo">FOO</label> <input type="text" id="foo"
 name="foo"\/>
<div tal:replace="structure python:ajax.checkedField(
      here.portal_url() + '/dval', 'foo', error='error')"/>

which might generate something like

<label for="foo">FOO</label> <input id="foo" name="foo" type="text">
<script type="text/javascript">
  new Form.Element.EventObserver($('foo'),
function(element, value) { new Ajax.Updater({success:
                                          $('error'),failure:
                                          $('error')},
                                          'http://site/bling/dval', {
onFailure:function(request){
      new Effect.Appear($('error'),{}); Fie\ld.activate('foo');},
parameters:Form.Element.serialize($('foo')),
onSuccess:function(request){new Effect.DropOut($('error'),{})},
asynchronous:true,insertion:null,method:'post\'} )  })

Fun Prototype callouts you didn't have to write or think about. Most of the kinds of tags you'd expect in rails are here already and I've added a few more to show how something like Plone could carry a standard array of reuable behaviors and visual effects.

Next we change something like Archetypes that already has per-field validation logic and vocabulary support and allow for things like auto-completion and validition from the widget layer with out change.

To show how simple this is I've written it into a product already. This will install the proper scripts into the resource registries in Plone. Just create a new site, install this product and play with the test.pt template in its skin path. The methods of interest are in Bling/ajax.py

To get the bundle you need darcs:

darcs get http://darcs.objectrealms.net/repos/Bling

or you can just look at the repo at
http://darcs.objectrealms.net/darcsweb.cgi/bling