Comments on: A co-server for Zope

By: Tarek Ziadé

Tarek Ziadé — Thu, 04 Oct 2007 02:52:21 +0000

@Jean-Nicolas Bès:

for the three first questions:

- the task can be a pure python independant function. in this case,
data is prepared, sent to the co-server, then the results are made available for the server to get them.

- the task is Zope dependant. In this case, the task is a generic task
that is called with the coordinates in the ZODB of the real task. This
generic task will know how to open the ZODB, and invoke the real task.
The real task can therefore apply the results directly in the ZODB.
In this case, the task acts a bit like a ZEO client yes. See how ZAsync does.

About the poll:

- each co-server needs to get the jobs by reading the DB yes,
why this would be a problem ?
- the job scheduling is a set of parameters in the SQL yes, then
each co-server scans the jobs and run them if it’s the right time.

Thanks for you feedback !

@Tiberiu: thanks for the extra infos, I need to digg into all those projects

@Jodok: Thanks for the explanation. If I get it well, you run the tasks in a separate process that is created from Zope itself. This means that Zope is responsible for task launching and managing, which is a choice. I want a separated task service for the reasons I mentioned, but i guess lovely.remotetask will fit for “normal” usages.

By: Jodok

Jodok — Tue, 02 Oct 2007 20:42:30 +0000

oops forget the last two sentences – they were below my scrollback

By: Jodok

Jodok — Tue, 02 Oct 2007 20:41:37 +0000

hi tarek,
it took 4 days until i read your blog post but here’s my feedback…
probably there is a misunderstanding about using XML-RPC… we don’t use XML-RPC to create remotetasks. you _can_ use xml-rpc to create remotetask (by creating a xmlrpc view that is generating it).
if you take a look at the readme (http://tinyurl.com/33xyga) you simply create a remotetask utility and create a task object.

the second issue – deploying an entire zope instance – that’s not necessary as well. lovely.remotetask spawns own processes for the taskservice – the tasks itself are beeing processed by creating a BaseRequest and using the publisher (publisher.publish). that means that you can use that instance for “normal” operation as well.
btw. in our setups we find that the remotetask instance has much more load than the other servers…

last thing – storage: the tasks itself are really small. if you process several thousand tasks per day it’s absolutely no problem to store them in zodb. we’re using zc.queue to store the queues on the zeo server. if the python packages of zc.queue and it’s objects are available on the zeo server as well – it will even do conflict resolution on the zeo server itself…

to be honest – i don’t want to switch to a sql-base remotetask service

for small installations it’s definitely an overkill to have dedicated instances for remotetasks

first of all

By: Tiberiu Ichim

Tiberiu Ichim — Tue, 02 Oct 2007 12:09:57 +0000

I haven’t used Hadoop, just read an article and a description on how to interact with python software
http://www.michael-noll.com/wiki/Writing_An_Hadoop_MapReduce_Program_In_Python

I can also think of some other ways to integrate external “dumb” applications with zope: for example, integrating them with an intelligent message passing system, such as Cabochon or ActiveMQ

As for real, personal experience with such a design, I’ve only worked with lovely.remotetask, but keep an eye on the alternatives.

http://www.openplans.org/projects/cabochon
http://activemq.apache.org/index.html

By: Jean-Nicolas Bès

Jean-Nicolas Bès — Mon, 01 Oct 2007 13:38:31 +0000

I like your “perfect” co-server design. I basically thought of something similar, but I have some questions that stay unanswered :
– From the co-server pov. What should we do about tasks input and output?
– From the ZODB pov. Who is reponsible for getting/putting data from/to ZODB?
– Could we imagine the co-server being a ZEO client where needed?
– Are the co-servers supposed to poll the SQL database for new jobs? (Main problem of a SQLdb-centric design IMHO)
– Where could we implement a specific task scheduling? Statically in the sqldb python API, letting the tasks consumer pick the right task?

Anyway, thanks a lot for this little “state of the art” on co-servers.

By: Tarek Ziadé

Tarek Ziadé — Sun, 30 Sep 2007 17:52:39 +0000

@Tiberiu : very interested, I didn’t know about it, have you tried it ?

By: Tiberiu Ichim

Tiberiu Ichim — Sun, 30 Sep 2007 16:37:23 +0000

Hadoop looks also very interesting as a potential Zope co-server. Though you’re probably already familiar with it. http://lucene.apache.org/hadoop/