Tag Archives: programming

zodb export import from zeo debug prompt – the simple built in way !

I am beginning to enjoy working with zodb. Again this is a quick note, pretty much like my previous post on zodb undo and conflict resolution..

I knew from my early Zope days that one could export and import arbitrary zope objects but never had a reason to code it or inspect its innards. Today though I wanted to export only a sub-section of our new beta site so that we could merge its contents with an already up and running ‘live’ site. So, I decided to search. One of the biggest problems with zope (and it’s related components) is the lack of authoritative and comprehensive documentation.

I found various scripts on the interwebs which showed how to walk a zodb object hierarchy. For instance:

However, there was very less information on how to actually export arbitrary objects using some built-in api (after all there had to be one !) and every less how to import exported .zexp files ! So, I hit the code. ZODB has a nicely named module ExportImport.py which exposes two neat functions exportFile() and importFile(). They do what they say and you’ll see the code is pretty much what you’d expect — walk the oids and serialize/load the objects …but wait there is more ! ZODB Connection class actually derive from ExportImport ! So, these functions may be used quite simply like this:

>>> # ...assuming you are in the debug prompt with context as the
>>> # object you want to export
>>> oid = context._p_oid
>>> conn = context._p_jar
>>> conn.exportFile(oid, 'context.zexp')
>>> # ...assuming you are in the debug prompt, you have an exported
>>> # zexp and context is the parent where you want to import and
>>> # place your imported 'new_folder'
>>> conn = context._p_jar
>>> imported = conn.importFile('new_folder.zexp')
>>> context['new_folder'] = imported
>>> import transaction
>>> transaction.commit()    # Do not forget to do this

Simple innit ? Hope someone out there finds this useful.

zodb undo and conflict resolution

This is going to be a quick post without much by way of explanation. It’s more for my own reference later (I tend to forget stuff I haven’t used in a while …or rather, I tend to forget …everything).

Last Friday a freak accident caused part of openideo.com disappear. I won’t get into the details but basically we needed to undo a delete operation on one of the top-level objects. Although we did have backup, it was a few hours old and with a site like openideo, a few hours translates to a lot of data. Now, I haven’t really had a reason to do a zodb undo from a script/debug prompt ever, so, I was really quite clueless. Also, the resident ‘all-things-zope’ guy was on leave so perfect friday night disaster. Well, I’ll cut to the chase. Here’s how simple zodb undo really is (note: code written for clarity not efficiency):

import time
import transaction
from ZODB import FileStorage, DB

# init the storage object
storage = FileStorage.FileStorage('Data.fs')

# create a db object
db = DB(storage)

# now, zope records all transactions in a log which is available as
# the db.undoLog(). Note that at the time of this writing the zope
# documentation is incorrect (bug report:
# https://bugs.launchpad.net/zodb/+bug/622828). The first and second
# parameters are the index within the list of undoable transactions,
# instead of time in seconds since epoch.
log = db.undoLog(0, sys.maxint)

# We wanted to undo transactions based on the time, since we knew
# exactly when the delete occured. However, undo-ing based on time is
# not necessary. undoLog() returns back a list of dicts with the keys,
# 'id' (unique identifier for the transaction), 'time' (time measured
# in seconds since epoch), 'description' (the .description attribute
# of the transaction) and 'user' (the .user attribute of the
# transaction). We could use any of these keys.

t = time.mktime((2010, 8, 20, 15, 12, 00, 00, 00, 00))
undo_list = []
for i in log:
    if i['time'] >= t:

for i in range(len(undo_list)): # Doing it this way so that if for
    tid = undo_list.pop(0)      # some reason the undo fails, we'd at
    db.undo(tid)                # least know which ones did not go
    transaction.commit()        # through

Now, while I was there I also came across this bit about resolving conflicts during commits. I have seen this ConflictError occur a number of times on openideo.com (coincidentally, in sections of code quite similar in intent to the hit counter example mentioned in the document). Fortunately though, these things don’t happen in a critical section of the site nor do they break it in any manner. So seeing this is more of an annoyance than a problem. I intend to try out that bit when I get the time. For now however, learning that the document is way back from 2002[1] and is not likely to be correct, I am less motivated to spend time researching whether that solution works.

If anyone knows for sure that this solution works for resolving conflicts please leave a note in the comments. Thanks !

[1] https://mail.zope.org/pipermail/zodb-dev/2010-August/013594.html

On reading code efficiently

As someone who learned to program the non-traditional way (ie: by reading and hacking on already written code) rather than ground up (ie: by writing hello world and progressing from there on), I feel the ability to read code is often under-stated when people offer advice on good programming practices. Especially considering that any good programmer, in her entire programming career would end up reading more lines of code than writing !

I have thought a bit about this subject (and also observed good programmers read code). While there are many tools which can be used to efficiently read code, I’d like to share my thoughts on the methodology.

Good programmers read code the same way most of us read a newspaper

  • They’ll look at the main entry and exit bits (the headlines) to get a ‘feel’ or overview of the nature of the application. This might also involve reading any design docs or UML diagrams or even badly drawn out bunch of boxes. The better that people are at looking and understanding the big picture, the less they have to look at to get it.
  • After that, they might jump into the bits that interest them (in our analogy — the sports section, the comics or *shudder* page 3). So, coders interested in just using the API will start exploring the headers. Those interested in hacking on the code, will start looking at the source files that are possibly named according their area of interest (that’s why naming and source file layout also is important while writing code). This is where cross-referencing tools help.
  • When looking at the code itself, for example, when tracing through a function, some bits are more or less taken at face value (as we do with phrases like ‘from reliable sources’ or ‘an independent study’) …at least during the first reading. In other words, faced with a statement like this:
    devices = probe_for_devices(args)
    …good programmers would only dive-into the definition of probe_for_devices() after they have a general enough understanding of why doing that is necessary in the caller (unless of course, /that/ function was what they were interested in looking at all along)
  • Now, when the time comes to not just passively reading but /interacting/ with the code, good programmers would first look for tell-tales signs on making such interactions easy (ie: what kind of logging/debugging facilities does the code provide ? can sub-components be built and tested independently ? Are there any compile time options available to assist this effort ? Can we get an intern to figure all this stuff out[1] …etc)

Doing all of the above of course is fun only if (again like natural written language), the quality of the code you are reading is good. Bad written code is also badly read code.

Now all that said, knowing how to use your tools well (ie: your editor, code cross-referencing tool, the man pages, API docs ..etc) improves the efficiency with which you read /immensely/. That’s how good programmers are able to ‘walk’ through the source in the same manner as the system would during execution.

FWIW: My tools of choice are, vim, cscope and a decently fast internet connection

Thanks for reading. Your views and comments are welcome.

[1] That’s where I usually came in šŸ™‚

Peeking into the linux loader

This post is as much as a tip as a demonstration on how beautifully simple it is to “look-neath-the-hood” when working with linux. This is the reason why linux programmers are happy programmers. Anyways, on with the details.
If you have ever seen errors like …

error while loading shared libraries: foo.so: cannot load shared object file: No such file or directory

when you know that foo.so exists, or have see something like ….

java.lang.UnsatisfiedLinkError: no foobar in shared library path
java.lang.UnsatisfiedLinkError: /some/module/soandso.so: libstdc++.so.6: cannot handle TLS data

you may use the LD_DEBUG feature to zoom-in on the problem.

The linux loader (/lib/ld-<version>.so) looks at the environment variable “LD_DEBUG” to print out interesting debug information as it loads libraries and resolves symbols. What exactly is printed out depends on the value that the variable is set to. You may get the list of valid values by setting LD_DEBUG=help. For example:

$ LD_DEBUG=help ls
Valid options for the LD_DEBUG environment variable are:

libs display library search paths
reloc display relocation processing
files display progress for input file
symbols display symbol table processing
bindings display information about symbol binding
versions display version dependencies
all all previous options combined
statistics display relocation statistics
unused determined unused DSOs
help display this help message and exit

To direct the debugging output into a file instead of standard output
a filename can be specified using the LD_DEBUG_OUTPUT environment variable.

What you make of this info. and how much it helps depends on the specific problems you are facing but all in all this is a useful thing to know when you are facing ^library issues^. More detailed information about working/writing libraries on linux is available at the Program Library HOWTO