Monthly Archives: August 2010

zodb export import from zeo debug prompt – the simple built in way !

I am beginning to enjoy working with zodb. Again this is a quick note, pretty much like my previous post on zodb undo and conflict resolution..

I knew from my early Zope days that one could export and import arbitrary zope objects but never had a reason to code it or inspect its innards. Today though I wanted to export only a sub-section of our new beta site so that we could merge its contents with an already up and running ‘live’ site. So, I decided to search. One of the biggest problems with zope (and it’s related components) is the lack of authoritative and comprehensive documentation.

I found various scripts on the interwebs which showed how to walk a zodb object hierarchy. For instance:

However, there was very less information on how to actually export arbitrary objects using some built-in api (after all there had to be one !) and every less how to import exported .zexp files ! So, I hit the code. ZODB has a nicely named module ExportImport.py which exposes two neat functions exportFile() and importFile(). They do what they say and you’ll see the code is pretty much what you’d expect — walk the oids and serialize/load the objects …but wait there is more ! ZODB Connection class actually derive from ExportImport ! So, these functions may be used quite simply like this:

>>> # ...assuming you are in the debug prompt with context as the
>>> # object you want to export
>>>
>>> oid = context._p_oid
>>> conn = context._p_jar
>>> conn.exportFile(oid, 'context.zexp')
.....
>>> # ...assuming you are in the debug prompt, you have an exported
>>> # zexp and context is the parent where you want to import and
>>> # place your imported 'new_folder'
>>>
>>> conn = context._p_jar
>>> imported = conn.importFile('new_folder.zexp')
>>> context['new_folder'] = imported
>>> import transaction
>>> transaction.commit()    # Do not forget to do this

Simple innit ? Hope someone out there finds this useful.

zodb undo and conflict resolution

This is going to be a quick post without much by way of explanation. It’s more for my own reference later (I tend to forget stuff I haven’t used in a while …or rather, I tend to forget …everything).

Last Friday a freak accident caused part of openideo.com disappear. I won’t get into the details but basically we needed to undo a delete operation on one of the top-level objects. Although we did have backup, it was a few hours old and with a site like openideo, a few hours translates to a lot of data. Now, I haven’t really had a reason to do a zodb undo from a script/debug prompt ever, so, I was really quite clueless. Also, the resident ‘all-things-zope’ guy was on leave so perfect friday night disaster. Well, I’ll cut to the chase. Here’s how simple zodb undo really is (note: code written for clarity not efficiency):


import time
import transaction
from ZODB import FileStorage, DB

# init the storage object
storage = FileStorage.FileStorage('Data.fs')

# create a db object
db = DB(storage)

# now, zope records all transactions in a log which is available as
# the db.undoLog(). Note that at the time of this writing the zope
# documentation is incorrect (bug report:
# https://bugs.launchpad.net/zodb/+bug/622828). The first and second
# parameters are the index within the list of undoable transactions,
# instead of time in seconds since epoch.
log = db.undoLog(0, sys.maxint)

# We wanted to undo transactions based on the time, since we knew
# exactly when the delete occured. However, undo-ing based on time is
# not necessary. undoLog() returns back a list of dicts with the keys,
# 'id' (unique identifier for the transaction), 'time' (time measured
# in seconds since epoch), 'description' (the .description attribute
# of the transaction) and 'user' (the .user attribute of the
# transaction). We could use any of these keys.

t = time.mktime((2010, 8, 20, 15, 12, 00, 00, 00, 00))
undo_list = []
for i in log:
    if i['time'] >= t:
        undo_list.append(i['id'])

for i in range(len(undo_list)): # Doing it this way so that if for
    tid = undo_list.pop(0)      # some reason the undo fails, we'd at
    db.undo(tid)                # least know which ones did not go
    transaction.commit()        # through

Now, while I was there I also came across this bit about resolving conflicts during commits. I have seen this ConflictError occur a number of times on openideo.com (coincidentally, in sections of code quite similar in intent to the hit counter example mentioned in the document). Fortunately though, these things don’t happen in a critical section of the site nor do they break it in any manner. So seeing this is more of an annoyance than a problem. I intend to try out that bit when I get the time. For now however, learning that the document is way back from 2002[1] and is not likely to be correct, I am less motivated to spend time researching whether that solution works.

If anyone knows for sure that this solution works for resolving conflicts please leave a note in the comments. Thanks !

[1] https://mail.zope.org/pipermail/zodb-dev/2010-August/013594.html