A few days ago I came across Feature #305877. What is it about? Well, Debian has the Popularity Contest, which tracks installed packages, how often they are used and sends an anonymized report once a week to their server. This maps the usage of Debian packages and as a nice side effect Debian guys can estimate the size of their user base on various platforms and releases. This also gives information about the community structure (e.g. how many users use development tools or graphic applications). This would be a very neat thing to have in openSUSE too!

At first, the task seemed pretty straightforward – just to replace dpkg calls with corresponding calls to rpm. There was one catch, though. Because of the transactions, which RPM uses, scanning on my openSUSE 11.1 machine took 2 minutes instead of 2 seconds on Debian! That’s because RPM creates one transaction for each package and constant locking and unlocking of rpmdb makes this process really slow. I rewrote the script to python, just to see how long will it take using only one long transaction and was very pleasantly surprised that it got back to 2 seconds. Moreover, rpmdb can tell you the exact time when the package was installed, so there was no need to check for ctime for files inside the packages like Debian does. (We still have to check for files atime to determine whether the package is used or not, though).

For the server part I was pretty sure about writing it in C to have it very fast and responsive, because I want to process incoming requests on the fly. The problem was with the storage. At the beginning I thought about using SQLite, but after some testing I decided to use much lighter disk-based hashtables TDB from the Samba team, because they perfectly fitted my humble needs.

Has this caught your interest? You can dig through the code at gitorious and any help is deeply welcome!. Yes, and why popcorn? Because the original is called popcon, but everybody at work just kept calling it popcorn during the discussions. Later I found another reason: popcorn is intended for RPM packages, so we definitively need an extra R in the name.

In the meantime, my colleague and Java packager Michal Vyskocil encountered a problem: it is very hard to find out which package installs particular Java class. Standa Brabec suggested that we could also process autoconf macros stored in m4 files. So we decided to merge these ideas together and we started the “Scout” project.

It will be a command line utility which will index various attributes of the packages and will allow the users to search in them. Each functionality will have its own module, so implementation could differ (we wanted the binary module to use SAT solver files and the others SQlite). I think that you’ll get the idea from the following picture:

Michal and I started development by creating a GIT repository (not much to see there, yet). Obviously this program will not appear in openSUSE 11.0, but we’d like to see it in 11.1 (and my plan is that command-not-found will use scout as its helper). At start, we will create 3 modules (binaries, java and autoconf), but later we’ll extend the support for python/ruby programmers.

If you have any ideas, do not hesitate and contact me!

Background: Debian and Ubuntu use patched bash, that allows to write handler, which is executed before “command not found” is written. That provides us a way we can help user in solving the problem. You can look at the picture to get the idea:

In first example, user types “epihpany” instead of “epiphany” and handler suggests correct spelling. After this user tries running epiphany, but it is not installed, so handler shows list of packages that provide this program. In the third example, user runs program that is not found, but package providing it is installed (meaning that program is not in user’s path – probably intended to be run only by root).

Package command-not-found and patched bash are available in BuildService or you can download packages directly from repository. Please test the packages and let me know if you have any ideas or in case you have found a bug.

Ah, I almost forgot You have to add these 3 lines to /etc/bash.bashrc.local or ~/.bashrc:

if [ -f /etc/bash_command_not_found ]; then
  . /etc/bash_command_not_found
fi

Handler for zsh can be installed in similar fashion. Read README in command-not-found package.