Lars Lohn

Syndicate content
Confessions of a Python Nut Case
Updated: 4 years 8 weeks ago

it’s a geeky meme

Sun, 04/13/2008 - 7:46pm

lars@bozeman:~$ history|awk ‘{a[$2]++} END{for(i in a){printf “%5d\t%s\n”,a[i],i}}’|sort -rn|head
133 cd
114 ls
44 svn
31 vi
28 python
24 ssh
21 ./ConfigurationManager.py
17 make
13 rsync

It looks to me like I spend too much time moving around the file system. I should try to type more pathnames and stick around in one place…

Categories: Planet OSL

Sanity Compromised by Firefox and ssh X Forwarding

Thu, 03/06/2008 - 8:48pm

Try this in Linux: open Firefox on your local machine. Then open a terminal window and ssh to another machine using the -X option for X forwarding. On the remote machine, start Firefox. The behavior I get is so bizarre that it cannot be a bug — somehow this looks intentional. The Firefox process on the remote machine sits for a few moments and then dies. Then a new local Firefox window opens. WTF?

I thought I was going insane. The people at the OSL that I told about this thought I was insane. The Mozilla developers that I work with and tried to explain this to thought I was insane.

Some research shows this: the remote Firefox actually starts and communicates with the X server running on the local machine. The X server tells the remote Firefox that there is already a process called Firefox running. The remote Firefox then sends a message to the local one to open a new window and then the remote Firefox dies. This protects a user from creating too many instances of a Firefox process on their machine. Clever, huh? But totally WRONG and counterintuitive!

Apparently you can stop this behavior if you start the remote Firefox with the intuitively named “no-remote” switch. That prevents the remote Firefox from “connecting” to the local Firefox.

Sigh, there goes an hour of my life that I’d like to have back…

Categories: Planet OSL

Python Generators: Searching Java Jar Files

Sun, 03/02/2008 - 7:02pm

Here is an example of a utility that uses a recursive generator. It is a command line utility that assists Java programmers in finding missing classes. I wrote this script several pears ago when I was dragged kicking and screaming into a Java project.  The script recursively searches a directory tree for jar files. When it finds a jar file, it scans the file’s directory for the target Java class.

#!/usr/bin/python

import sys, os, os.path
import fnmatch

def findFileGenerator(rootDirectory, acceptanceFunction):
  for aCurrentDirectoryItem in [ os.path.join(rootDirectory, x) for x in os.listdir(rootDirectory) ]:
    if acceptanceFunction(aCurrentDirectoryItem):
      yield aCurrentDirectoryItem
    if os.path.isdir(aCurrentDirectoryItem):
      for aSubdirectoryItem in findFileGenerator(aCurrentDirectoryItem, acceptanceFunction):
        yield aSubdirectoryItem

if __name__ == "__main__":
  rootOfSearch = '.'
  if sys.argv[1:]:
    rootOfSearch = sys.argv[1]
  if sys.argv[2:]:
    classnameFragment = sys.argv[2].replace('.', '/')
    def anAcceptanceFunction (itemToTest):
      return not os.path.isdir(itemToTest) and fnmatch.fnmatch(itemToTest, '*.jar') and
             classnameFragment in os.popen('jar -tf %s' % itemToTest).read()
  else:
    def anAcceptanceFunction (itemToTest):
      return not os.path.isdir(itemToTest) and fnmatch.fnmatch(itemToTest, '*.jar')

  try:
    for x in findFileGenerator(rootOfSearch, anAcceptanceFunction):
      print x
  except Exception, anException:
    print anException

The focus is on the generator function findFileGenerator. It creates an iterator for the results of a recursive search through a directory tree. It accepts as parameters a path to begin the search and a function to determine if a given file satisfies the search parameters.

Generators can be kind of confusing because even though they look like a function, they do not execute immediately when called. They return a reference to an object that works like an iterator. The code defined in the generator function is executed by that iterator object. The first time that the iterator’s ‘next’ function is called, execution begins at the beginning of the code and goes until it encounters a ‘yield’ statement. The ‘yield’ statement returns the next value of the iterator. The next time the ‘next’ function is called, execution resumes at the next statement after the ‘yield’.

Let’s examine this example closely. Imagine that the first call to the iterator has happened and we’ve got the resultant iterator-like object. The first call on that object to ‘next’ starts execution at this line:

for aCurrentDirectoryItem in [ os.path.join(rootDirectory, x) for x in os.listdir(rootDirectory) ]:

Here we’re getting our first directory listing of all the files in the current directory. Because the call to ‘os.listdir(rootDirectory)’ returns a list of file names with their paths stripped off, we’re going to have to re-attach them. The list comprehension (the code between the [ … ]) welds the current directory path to each of the files in the list and returns a new list. The for loop then sets us up to iterate through that list.

   if acceptanceFunction(aCurrentDirectoryItem):
      yield aCurrentDirectoryItem

Here’s where we decide if the current entry in this directory is interesting or not. We call the acceptance function on the item. Since the acceptance function is passed in when we originally called this generator, it could be anything the programmer desired. In the case of this particular utility, we’re looking for Java Jar files that meet a certain criteria. But it really could have been anything at all: find all files that have vowels in their name, or all files that have a specific type or content.
If the acceptance function returns ‘True’, then we yield. The current file is returned by the iterator and execution stops until the ‘next’ function is called.

  if os.path.isdir(aCurrentDirectoryItem):

If the acceptance function rejected the item, this is immediately the next line to execute. If the acceptance function accepted the item, this line won’t be called until after the next call to ‘next’. In either case, our goal is to find the next item for the iterator to return.

Since we’re iterating through a list of entries in a directory, some of those will be directories themselves. The item that we sent to the acceptance function could have been a subdirectory. Regardless of the outcome of the acceptance function, we need to recurse into subdirectories.

   for aSubdirectoryItem in findFileGenerator(aCurrentDirectoryItem, acceptanceFunction):
        yield aSubdirectoryItem

Hang onto your hat, here’s where your brain may explode. We’ve got a sub-directory and we need to recurse into it and iterate through its entries. Well, we’ve got this handy generator that does exact that: it returns an iterator that will cycle through the contents of directory. ‘for’ statements in Python have a special relationship with iterators. You can provide one instead of a list and the ‘for’ loop will dutifully iterate through them for you. We recursively call the generator, passing in the subdirectory and the acceptance function. The generator returns an iterator to us and the for statement starts the iteration by silently calling the next function. Remember that the iterator returns only items that have passed the acceptance function, so each item that we get here we’re just going to pass on as the next item in our iterator. Hence, we yield every item that we get in this loop.

The rest of the file is in the problem domain: a command line utility that will find Java Jar files with certain classes in them.

if __name__ == "__main__":

Perhaps someday in the future, we’ll want to use the generator in another application. By putting the code of the command line utility under this ‘if’, we’ll prevent it from executing when we use the ‘import’ statement on this file.

  rootOfSearch = '.'
  if sys.argv[1:]:
    rootOfSearch = sys.argv[1]

The root of the path that we’re to search is option on the command line. If no path is specified, we’ll assume that we’re to start in the current working directory.

  if sys.argv[2:]:
    classnameFragment = sys.argv[2].replace('.', '/')
    def anAcceptanceFunction (itemToTest):
      return not os.path.isdir(itemToTest) and fnmatch.fnmatch(itemToTest, '*.jar') and
             classnameFragment in os.popen('jar -tf %s' % itemToTest).read()

The name of the class that we’re to search for is also optional. If the user does not provide one, then we’ll assume that we’re to just find all jar files regardless of their content.

This code fragment is the other case: a fragment of a class name has been given. It is our task here to create an acceptance function that meets the criterion.

First thing to do is cook the class name a bit. In Java, class names are qualifies with paths. Inside Java code, ‘.’ is used as a separator. However, inside jar files, ‘/’ is the separator. To be friendly, we want Java programmers to be able to use either notation. We make sure the command line argument is converted to the ‘/’ notation and stored in ‘classnameFragment’. Next we define an acceptance function that receives a pathname as a parameter. All we have to do is subject that pathname to some tests and give it either a thumbs up or down. In this case, we test to see if the pathname represents a directory, then test to see if it is a jar file and finally we run the command line function ‘jar \-tf’ to give us a listing of the jar to see if our class name fragment is in there. Since Python can do “short-circuit” expression evaluation, if any of the earlier tests fail in the boolean expression, the other tests do not get executed.

  else:
    def anAcceptanceFunction (itemToTest):
      return not os.path.isdir(itemToTest) and fnmatch.fnmatch(itemToTest, '*.jar')

In the case where the user did not provide a class name fragment, we assume that we’re looking for all jar files. The acceptance function here just drops the additional criterion where we looking into the content of the jar file.

  try:
    for x in findFileGenerator(rootOfSearch, anAcceptanceFunction):
      print x
  except Exception, anException:
    print anException

Finally, we ‘re ready to actually use the tools. We call the generator function with the path from which to start the search and our acceptance function. That returns an iterator that we loop through and print the matching jar files.

Categories: Planet OSL