Category Archives: hacks you can live without

zodb export import from zeo debug prompt – the simple built in way !

I am beginning to enjoy working with zodb. Again this is a quick note, pretty much like my previous post on zodb undo and conflict resolution..

I knew from my early Zope days that one could export and import arbitrary zope objects but never had a reason to code it or inspect its innards. Today though I wanted to export only a sub-section of our new beta site so that we could merge its contents with an already up and running ‘live’ site. So, I decided to search. One of the biggest problems with zope (and it’s related components) is the lack of authoritative and comprehensive documentation.

I found various scripts on the interwebs which showed how to walk a zodb object hierarchy. For instance:

However, there was very less information on how to actually export arbitrary objects using some built-in api (after all there had to be one !) and every less how to import exported .zexp files ! So, I hit the code. ZODB has a nicely named module which exposes two neat functions exportFile() and importFile(). They do what they say and you’ll see the code is pretty much what you’d expect — walk the oids and serialize/load the objects …but wait there is more ! ZODB Connection class actually derive from ExportImport ! So, these functions may be used quite simply like this:

>>> # ...assuming you are in the debug prompt with context as the
>>> # object you want to export
>>> oid = context._p_oid
>>> conn = context._p_jar
>>> conn.exportFile(oid, 'context.zexp')
>>> # ...assuming you are in the debug prompt, you have an exported
>>> # zexp and context is the parent where you want to import and
>>> # place your imported 'new_folder'
>>> conn = context._p_jar
>>> imported = conn.importFile('new_folder.zexp')
>>> context['new_folder'] = imported
>>> import transaction
>>> transaction.commit()    # Do not forget to do this

Simple innit ? Hope someone out there finds this useful.

zodb undo and conflict resolution

This is going to be a quick post without much by way of explanation. It’s more for my own reference later (I tend to forget stuff I haven’t used in a while …or rather, I tend to forget …everything).

Last Friday a freak accident caused part of disappear. I won’t get into the details but basically we needed to undo a delete operation on one of the top-level objects. Although we did have backup, it was a few hours old and with a site like openideo, a few hours translates to a lot of data. Now, I haven’t really had a reason to do a zodb undo from a script/debug prompt ever, so, I was really quite clueless. Also, the resident ‘all-things-zope’ guy was on leave so perfect friday night disaster. Well, I’ll cut to the chase. Here’s how simple zodb undo really is (note: code written for clarity not efficiency):

import time
import transaction
from ZODB import FileStorage, DB

# init the storage object
storage = FileStorage.FileStorage('Data.fs')

# create a db object
db = DB(storage)

# now, zope records all transactions in a log which is available as
# the db.undoLog(). Note that at the time of this writing the zope
# documentation is incorrect (bug report:
# The first and second
# parameters are the index within the list of undoable transactions,
# instead of time in seconds since epoch.
log = db.undoLog(0, sys.maxint)

# We wanted to undo transactions based on the time, since we knew
# exactly when the delete occured. However, undo-ing based on time is
# not necessary. undoLog() returns back a list of dicts with the keys,
# 'id' (unique identifier for the transaction), 'time' (time measured
# in seconds since epoch), 'description' (the .description attribute
# of the transaction) and 'user' (the .user attribute of the
# transaction). We could use any of these keys.

t = time.mktime((2010, 8, 20, 15, 12, 00, 00, 00, 00))
undo_list = []
for i in log:
    if i['time'] >= t:

for i in range(len(undo_list)): # Doing it this way so that if for
    tid = undo_list.pop(0)      # some reason the undo fails, we'd at
    db.undo(tid)                # least know which ones did not go
    transaction.commit()        # through

Now, while I was there I also came across this bit about resolving conflicts during commits. I have seen this ConflictError occur a number of times on (coincidentally, in sections of code quite similar in intent to the hit counter example mentioned in the document). Fortunately though, these things don’t happen in a critical section of the site nor do they break it in any manner. So seeing this is more of an annoyance than a problem. I intend to try out that bit when I get the time. For now however, learning that the document is way back from 2002[1] and is not likely to be correct, I am less motivated to spend time researching whether that solution works.

If anyone knows for sure that this solution works for resolving conflicts please leave a note in the comments. Thanks !


vim backups using git

I am a vim guy. I’ve been hooked on to it ever since my first introduction to Unix (back in ~1995). One of the things I like about vim is that it is just an editor. Nothing more, nothing less. It does allow for creation of plugins, but only as far as they complement editing tasks. So, although you’ll find vim plugin scripts that provide file browsers, IDE features, encryption etc in vim, you will almost certainly never find an integrated mail client, IRC client, web browser etc (things which are rumoured to exist for a long time in that other editor which shall not be named).

The reason I mention this is because, I recently came across this interesting plugin named VimLocalHistory which maintains a backup of files that you edit in vim using git at every write to the file. Cool, ehe ? btw, git is another really cool tool that I’ve fallen in love with recently but more of that in another post probably.

The second reason for this post is, considering how i have waste^H^H^H^H^Hspent a lot of time creating, maintaining and tweaking my preferred vim configuration and plugins(*), it only makes sense that I share it with friends who share my preferences and style of working but don’t/can’t look for the tools that’ll help them at it . So, here is my vim configuration:

Obviously, you’d have to rename dot_vimrc and dot_vim to ~/.vim and ~/.vimrc respectively. The dot_vimrc file is well commented (i think) so reading that and adopting it to suit your needs should not be a problem. Enjoy.

(*) I created the file in 2004, IIRC and have maintained it ever since

Reblog this post [with Zemanta]

On reading code efficiently

As someone who learned to program the non-traditional way (ie: by reading and hacking on already written code) rather than ground up (ie: by writing hello world and progressing from there on), I feel the ability to read code is often under-stated when people offer advice on good programming practices. Especially considering that any good programmer, in her entire programming career would end up reading more lines of code than writing !

I have thought a bit about this subject (and also observed good programmers read code). While there are many tools which can be used to efficiently read code, I’d like to share my thoughts on the methodology.

Good programmers read code the same way most of us read a newspaper

  • They’ll look at the main entry and exit bits (the headlines) to get a ‘feel’ or overview of the nature of the application. This might also involve reading any design docs or UML diagrams or even badly drawn out bunch of boxes. The better that people are at looking and understanding the big picture, the less they have to look at to get it.
  • After that, they might jump into the bits that interest them (in our analogy — the sports section, the comics or *shudder* page 3). So, coders interested in just using the API will start exploring the headers. Those interested in hacking on the code, will start looking at the source files that are possibly named according their area of interest (that’s why naming and source file layout also is important while writing code). This is where cross-referencing tools help.
  • When looking at the code itself, for example, when tracing through a function, some bits are more or less taken at face value (as we do with phrases like ‘from reliable sources’ or ‘an independent study’) …at least during the first reading. In other words, faced with a statement like this:
    devices = probe_for_devices(args)
    …good programmers would only dive-into the definition of probe_for_devices() after they have a general enough understanding of why doing that is necessary in the caller (unless of course, /that/ function was what they were interested in looking at all along)
  • Now, when the time comes to not just passively reading but /interacting/ with the code, good programmers would first look for tell-tales signs on making such interactions easy (ie: what kind of logging/debugging facilities does the code provide ? can sub-components be built and tested independently ? Are there any compile time options available to assist this effort ? Can we get an intern to figure all this stuff out[1] …etc)

Doing all of the above of course is fun only if (again like natural written language), the quality of the code you are reading is good. Bad written code is also badly read code.

Now all that said, knowing how to use your tools well (ie: your editor, code cross-referencing tool, the man pages, API docs ..etc) improves the efficiency with which you read /immensely/. That’s how good programmers are able to ‘walk’ through the source in the same manner as the system would during execution.

FWIW: My tools of choice are, vim, cscope and a decently fast internet connection

Thanks for reading. Your views and comments are welcome.

[1] That’s where I usually came in šŸ™‚

How I helped the office chef

If you know me personally (why would you be reading this otherwise ?), it’s quite possible that you are thinking about some smart-ass remark to make in the comments after reading the title …or maybe you are just plain shocked. Well, relax, you may breathe normally now. I did not help the chef with his work. I helped him with the only thing that I am good at …well, at least kinda good at — tech stuff.

The chef at our office is a charming Pakistani chap who likes speaking in Hindi whenever he sees me (inspite of the quality of hindi he hears me speak !). So, we tend to indulge in small talk as he serves me my lunch.

Sometime last week, he asks me whether I can help him with a computer problem (don’t they all at some point šŸ™‚ !?!), and nice fella that I am, I say yes. He goes back into his kitchen (or whatever that large room that is, which is always kept closed; away from the view of the good people eating their lunches) and brings back a fairly old Compaq laptop.

Says, it doesn’t work (hmm, nice problem description). I power it up and don’t even have to ask him to elaborate. I hear the all too familiar, click-tik-tik-tik-click-tik… of the Hard disk.

So, i tell him — Aapki disk mar gayee.

Fortunately, he actually knows what I am talking about (so, he doesn’t have to suffer more tech talk in hindi). He says, he showed it to some other techies in the office and all of them said, there is nothing that can be done except for buying a new harddisk. (most of these other techies are windoze people). He was hoping I could do some magic …all he wants is to access the net, play movies and songs. He has a desktop at home, so any other job (like his son’s school work) can be done on that.

That’s when i realize (or perhaps remember ?), it is so ridiculous that an broken hard disk equates an unusable system — if you live in a proprietary software world, that is.

I tell him, I can certainly help him, if he has a USB stick and is ready to learn something new. He was all for it ! (another observation — most non-tech people don’t know nor do they care what an OS is, it is all the same to them — until of course they actually see something different, like a system boot up from a USB stick, without a hard disk attached).

So, he gives me his 4GB USB stick and offers the laptop too. I say, I don’t need it. He gives me a confused look, but I can see, he is really hoping for magic.

I won’t get into the details (since, it isn’t deep magic really, in fact people like Luke an Kushal have made it downright boring), I’ll just list the steps:

  • get the latest and greatest Fedora release (Fedora 10, right now)
  • get the liveusb-creator
  • run liveusb-creator
  • press two buttons, i’m ain’t telling you which, if you have to ask
  • boot system with new live USB install
  • configure the system for net access
  • add the repos rpmfusion-free, rpmfusion-nonfree, adobe (for flash plugin) and freshrpms (for libdvdcss)
  • install mplayer, libdvdcss, ffmpeg, vlc, audacious, gecko-mediaplayer, flash-plugin
  • (note: my aim was to give the chef a usable system)

Well, all that done, I gave the USB stick to the chef and asked him to boot his laptop using it. I could see, he was still trying to understand what was happening, especially since i was idly fiddling with the br0ke Hard Disk in my hand.

The look on his face, when he saw a login screen is indescribable ! I showed him the way around the GNOME UI. Got him connected to the wireless, got him playing movies, songs and saving files off the net. He then tells me why he has that ear to ear grin.

Apparently, the hard disk had been misbehaving for quite sometime. Random hangs followed by multiple tries to get the system booting, followed by the eventual clicking sounds. It finally stopped working about 3 months ago. He took it first to the laptop vendor, where he found out that getting a new hard disk was kinda expensive. Since his computing needs were limited, he didn’t think it wise to buy a disk here. He had planned to buy the disk the next time he visited Pakistan (since computer spares are supposed to be cheaper there) and had resigned to the fact that the laptop was dead.

However, out of sheer coincidence (I assume, it was more like running out of topics for small talk), he asked me to help since I was a techie …and I say, I can …and I do …and all in a matter of a few hours !! He just couldn’t get over the fact.

Now, he is telling everyone he is chatty with (and that’s a *lot* of people, including especially, the ‘other’ techies who had a go at the laptop) about this guy who made his laptop run without a hard disk.

Well, what can I say !?

Thanks Luke, Kushal and all the guys working on Fedora …and thanks to FOSS developers all over the world ! …it’s because of you guys, at least one guy thinks I am some sort of a magician.