stick's corner

Today I had a very nice surprise laying on my office desk. It was a postcard from my dear friends from Brno (so called #fedora-cs mafia ) depicting a 3D image of a geeko on sand. Luckily I was able to find two shops (here and here) which offer this jewel. The latter one also had the 3D picture so I can share this viewing pleasure with you (click on it to animate):

It looks even better in real-life!

I use Mozilla Thunderbird as my e-mail client and I prefer plain-text messages over HTML format. When you view a threaded conversation it looks like this:

I found a little hack on Mozilla website which adds different colors for various quote levels in messages. Unfortunately the example is no longer valid for Thunderbird 3, because the default color scheme had changed. I modified the CSS to fit the new look and ended up with this:

If you like it, just run the following code to have it applied:

cd ~/.thunderbird/*.default
mkdir chrome
cat <<EOF > chrome/userContent.css
/* Quote Levels Colors */
/* bar color: #729fcf */
blockquote[type=cite] {
    color: #394f67 !important;
    background-color: #edf3f9 !important;
}
/* bar color: #ad7fa8 */
blockquote[type=cite] blockquote {
    color: #563f54 !important;
    background-color: #f4eff4 !important;
}
/* bar color: #8ae234 */
blockquote[type=cite] blockquote blockquote {
    color: #45711a !important;
    background-color: #f0fbe5 !important;
}
/* bar color: #fcaf3e */
blockquote[type=cite] blockquote blockquote blockquote {
    color: #7e571f !important;
    background-color: #fef5e6 !important;
}
/* bar color: #e9b96e */
blockquote[type=cite] blockquote blockquote blockquote blockquote {
    color: #745c37 !important;
    background-color: #fcf6ec !important;
}
EOF

Don’t forget to restart Thunderbird!

If you are closely following Ruby development and especially the situation around ruby gems, you might already know of Gemcutter. It is a new service, which provides a very easy way how to publish gems and also a good API to deal with them. It is not trying to replace RubyForge as whole, just its gem hosting (+ now defunct GitHub gem hosting) and will soon become the central and the only place for Ruby gems. The whole site is MIT licensed and the code is available on GitHub.

During the winter holidays I wrote a simple script which utilizes the Gemcutter API and prints versions of rubygem-* packages in our devel:languages:ruby:extensions Build Service repository compared with the corresponding gem versions on Gemcutter. Using this script and a great gem2rpm (more particularly gem2rpm-opensuse command which applies openSUSE template and is available from rubygem-gem2rpm package), I was able to update nearly a hundred of gems in just two hours. Rails rubygems have a specific packaging in openSUSE, so I left them out, but more than 90% of the rest didn’t need any changes in autogenerated spec file.

This brought me an idea. If only Gemcutter had an option to somehow send out notification that a new gem has been pushed, we could automate the process and have up-to-date rubygems in our devel:languages:ruby:extensions repository almost instantly. (We would still need to keep the list of “dirty” rubygems that need to be updated manually, though. For example, Rails packages I mentioned earlier, where we keep multiple versions, or others where we need to add a patch replacing /usr/local/bin/ruby with /usr/bin/ruby in scripts).

Few days later, Gemcutter gained RSS feed support, but only for the gems one is interested in. I didn’t find the option to have RSS feed for all gems. This could have helped in creating such mechanism, but that won’t be needed anymore because …

… yesterday Nick Quaranto of Gemcutter announced webhook support. I’m really excited, because that’s exactly what we need! When one registers a webhook, Gemcutter emits a POST request on a certain URL when a gem is pushed or updated. This request is a JSON document containing the info about gem. What we need is to create a mechanism that:

• receives notification via POST JSON request
• checks whether the package is not “dirty” → exit if it is (and probably send some email …)
• fetches the package from the Build Service or create a new one
• fetches the new gem, removes the old one
• runs gem2rpm-opensuse to create a spec file replacing the old one
• adds changelog entry
• pushes the updated package back into the Build Service

Last but not least: If Fedora and Mandriva had gem2rpm templates in a perfect shape too, Build Service could provide packaged gems also for their distributions.

So what do you think? Any volunteers for this? Right now, I’m off to fix some small bugs I found in gem2rpm while fiddling with it …

The middle of November was very exciting for both Fedora and openSUSE communities. At first, openSUSE project unleashed its 11.2 release, which was followed by Fedora 12 a few days after. I thought it would be interesting to dig into bug reports which were filed during the development of these two releases in respective bugzillas.

I’m not going to compare the absolute number of bugs, nor the ratio of reported/closed ones, because I think these statistics are easy to find. Also, Fedora 12 development took 6 months, while openSUSE 11.2 took 11 months and this is not very comparable. What I was interested in was how much work happens inside the companies and how much outside their walls, in the community. Please, bear in mind that development is not only about reporting, closing bugs and their count. A lot also happens on wikis, openFATE or other tools, so these statistics could be a little bit skewed. Enough talking, here comes the data and charts …

* – assignee could be mailing list, it does not have to be an individual

Chart 1: Bugs reported by

Chart 2: Unique reporters

When we look at the reporters charts, we see that they are quite similar. This is good!

Chart 3: Bugs assigned to

Chart 4: Unique assignees

The next two charts are where we can see drastic differences. What are the reasons for this? Well, I was able to come up with these:

• like mentioned above, openSUSE project utilizes other tools, e.g. openFATE, to steer development
• openSUSE is younger project than Fedora, so the community involvement is lower (sometimes it is still very hard for externals to understand WHAT and HOW should be achieved)
• Novell folks do not reassign bugs to community members, even if the problem is fixed via Build Service submit request, the bug stays assigned to ‘insider’ who closes it
• some of the assignees are in fact mailing lists that contain both internal and external people, but they belong to novell.com domain
• Fedora uses bugzilla for package reviews, these often include external people but are not actual bug reports

Can you think of any more reasons? What can we do to improve the situation and to engage openSUSE community more?

Lots of programs that bring their own libraries use the following snippet in their wrapper scripts:

export LD_LIBRARY_PATH="/my/special/librarypath:$LD_LIBRARY_PATH"

This allows linker to find the needed libraries, even if they are not located in the standard directories (which are defined by /etc/ld.so.conf). At first, this seems OK, but it creates one problem, though. When the $LD_LIBRARY_PATH was empty before the assignment, the new value ends with a colon. When we run the program wrapper, linker splits the variable into substrings and ends up with one empty path. This indicates to search for libraries in the CURRENT working directory, which can cause problems or even a security threat.

So, what’s the correct way of defining the library path? Of course, we could check if the variable is empty before the assignment like this:

if [ -n "$LD_LIBRARY_PATH" ]; then
    export LD_LIBRARY_PATH="/my/special/librarypath:$LD_LIBRARY_PATH"
else
    export LD_LIBRARY_PATH="/my/special/librarypath"
fi

but there is one neat shell trick we can use (should work on all POSIX shells). The description says:

${parameter:+word}
Use Alternate Value. If parameter is null or unset, nothing is substituted, otherwise the expansion of word is substituted.

In the end we have:

export LD_LIBRARY_PATH="/my/special/librarypath${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"

which is not much longer than the line we started with, but does the assignment correctly. (The first colon belongs to shell syntax, the second one is a part of the value being appended).

I’m sure you all heard about the openSUSE Conference 2009 that took place in September in Nuremberg. Not so many know about the RPM Summit that was a part of the conference during its first two days. Idea to create something like this started at LinuxTag 2009 when Zonker invited Florian to Nuremberg. One thing lead to another and in the end we managed to get several important people into one room (or to join us via conf call) and we kept them there until all problems with RPM and package management in general were solved. Well, not really, there was a lot to discuss and there still is, but you got the point.

And who were our victims^Wexperts? Here’s the list:

• Michael Schroeder, SUSE RPM maintainer (Novell)
• Florian Festi, RPM upstream developer (Red Hat)
• Panu Matilainen, RPM upstream developer (Red Hat)
• Klaus Kaempf and Stanislav Visnovsky, SUSE system management folks (including software management, e.g. libzypp) (Novell)
• Seth Vidal and James Antill of the YUM fame (Red Hat)

We were also joined by others interested in RPM development: Ludwig Nussel, Olaf Dabrunz and Jan Engelhardt. Most of the time we were working through a quite long list of issues we prepared in advance in the wiki and it turned out that there is not much difference in the view on the different topics. I tried to summarize the summit in the following 10 points:

1. Virtual triggers

   Triggers on virtual symbols (provides) do not work at the moment. This is considered a bug (or missing feature) and will be fixed.

2. File triggers

   Panu is working on file triggers feature. This will help packagers to get rid of a large number of scriptlets (e.g. %post/%postun -p /sbin/ldconfig) in spec files, but he wants to be sure that it doesn’t break anything else. File triggers are already used by Mandriva, but Panu wants to implement them in a different fashion. (See Mandriva wiki for the details about their implementation).

3. Soft dependencies

   Soft dependencies keywords that are already used by SUSE (Recommends, Suggests, Supplements, Enhances) will be recognized in the future versions of RPM. RPM will not do anything with them except of storing in the database and reporting to higher levels of package management stack. There is an ongoing discussion whether and how to implement soft dependencies in YUM (they are already implemented in the zypp stack).

4. Update scriptlets

   We agreed to introduce two new scriptlets called %preup and %postup. These will be called when updating the package (%preun, %postun, %pre and %post will not be run in this case). This is a way how to fix the broken uninstall scriptlets and one doesn’t have to do the weird if [ "$1" = "0" ]; stuff in the scriptlets anymore to detect whether the operation is upgrade or uninstall.

5. Scriptlets arguments

   Package scriptlets will get more information (e.g. NEVRA of the packages) about currently run transaction in environment variables. This will make it easier for scriptlets to handle special situations or to detect more precisely what is happening. (One could for example convert some config files when upgrading from package of version 1.x to 2.x and to leave them intact otherwise).

6. DeltaRPM

   DeltaRPM sources will stay in gitorious at the moment. They might appear in RPM upstream git repository later in the future but there is no reasoning for this right now. Jindrich Novy is working on a new diff algorithm, we’ll see if it will be merged into DeltaRPM or replace it completely.

7. New payload format

   Currently used format is CPIO but it has a filesize limit of 8 GiB. This is not enough when one tries to distribute very large files, for example appliance images, packaged as RPMs. Developers will need to find a way how to enhance CPIO or try another format soon. TAR is really not an option …

8. Tilde in version

   The special character tilde (~) will be available for use in version representing a negative version token. This will simplify complicated rules which abuse the version and the release tags. (For example, using foo-2.5.99.2 instead of the foo-2.6~beta2). There already exists a patch but it needs more test coverage.

9. Easy way to add or remove autogenerated dependencies

   This feature is wanted but is not implemented yet. Currently this is workarounded by redefining the __find_provides/__find_requires/… in spec file. Example from the open-vm-tools package:

   # exclude AMD PCnet32 LANCE pci.id from Supplements list [bnc#397554]
   %define __find_supplements sh -c '/usr/lib/rpm/find-supplements %{name} | grep -v pci:v00001022d00002000'

10. Logging and recovery

    There is a plan to include a transaction log to RPM which will allow recovery after a broken transaction. RPM log could also replace similar features in YUM/zypper and will be more reliable as it would also work from rpm cli.

That’s it! I had a great time during the conference and was really happy to see the synergy between the developers, even though they are working for two competing companies. In case you have any questions feel free to join us on Freenode IRC channel #rpm.org or use the appropriate mailing list.

Michael Schroeder has finally updated the RPM package in openSUSE to the latest upstream version 4.7.1. \o/ There were LOTS of bugfixes, enhancements and internal API changes which are probably not very interesting for mere mortals, but what do they mean for packagers? Here’s a list of important changes for them:

• Macro %patch does not behave like %patch0 anymore. Stop mixing Patch: with %patch0 and vice versa and use them consistently – i.e. either use the numbers in preamble and in %prep phase or don’t.
• Fuzz tolerance for patches was changed from 2 to zero. All patches must apply cleanly now.
• Sub-packages can be declared as “noarch” now. Such .spec file is incompatible with older RPM but the resulting binary packages are compatible.
• Section %files now supports multiple file lists. No need to join the files into one in %install section.
• The new macros %{patches} and %{sources} are available. You can use them in for loops to iterate over all patches and sources respectively.
• %ghost files do not have to be present in the build root anymore. In this case you have to specify their attributes like this: %attr(0644,root,root) %ghost ghostfile.

Few months ago we began discussion with Fedora guys about how to bring our packaging habits more closer. We identified some points by comparing their Packaging Guidelines with our routines and I created a list of differences at RPM wiki. The main idea is to find a solution which is acceptable by both parties and to create a mechanism (usually macros) for it in RPM upstream, so both distributions can benefit from it. Some of the problems are already sorted out, but there is still a long way to go. If you are aware of such differences or even would like to propose the solution, please contact me. I’ll add the items to the list and we’ll discuss them at the openSUSE Conference – RPM Summit, where people from RPM upstream will be present as well. By looking at the “Solved” part (at the bottom) of the mentioned wiki page you can see the following changes to be more compatible:

• Our %makeinstall macro is called %make_install in RPM upstream. Change it to the new form when you touch your package (also replace make DESTDIR=${buildroot} install with this macro).
• Build root is created in a safe way, you don’t need to specify the location with BuildRoot:. Also it is automatically cleaned after the build if the %clean phase is omitted from .spec file (i.e. not mentioned at all, if it is there but is empty no removal is done).
• Use %{python_sitelib} macro for noarch Python packages and %{python_sitearch} for the architecture dependent ones, instead of the old %{py_sitedir} one. (Yes you can now create noarch Python packages!) Stop using %{py_requires} as well, because dependencies are added automatically now.

Important! Nearly all of these changes are effective from 11.2 and will not work with earlier releases in openSUSE. If you plan to use the same .spec file for backporting inside BuildService, you’ll need to keep the old and the new stuff in corresponding %if 0?%{?suse_version} ... %endif blocks for a while (well, until 11.1 is EOL).

PS: I will try to keep this post updated until 11.2 is released so you might want to check it from time to time.