Natural Order Development

In Ubuntu there are a number of ways to get upgrades. There is the update manager which is configured to run automatically. There is using the command line approach with apt-get ('sudo apt-get update && sudo apt-get upgrade'). Finally there is the GUI utility called Synaptic ('sudo synaptic').

So what happens when something gets updated and introduces a bug, or breaks another applications behavior?

I had this happen to me just this week. I was running along great, my services were all happy. Then the update manager popped up saying there were some new upgrades. Being security minded I like to keep all my libraries at the most current level. Of course I looked over the list and there was an update to the ssl libraries. Nothing concerning in the release notes for the changes. So I let it get installed.

Some time later I finished changing some REST services, had a clean build and deployed to my testing Tomcat server. At which point my integration tests failed. Poking at the problem and turning up the verbosity got the error message:

SSLError: [Errno 8] _ssl.c:503: EOF occurred in violation of protocol

That was new. I hadn't changed the Python script that was producing this error message. Naturally I thought it was all the changes I had made. I Googled and Googled and read and found nothing even remotely helpful. OK so I then checked against the production services. OH MY they were producing the same error! Panic adrenalin is sometimes good sometimes not. In this case not. I spun and started calling other developers. They all checked and hey no issues. Hmm OK no issues, really? Alright then it must just be me. Wait a tick didn't I install some patch to SSL. Oh ya. OK check another machine that is not running that patch. Sure enough everything works great.

You'd think problem solved at this point. But, you'd be wrong. (Or maybe not, you are after all reading this, which means it is likely you are having a similar problem.)

Thus began the quest to downgrade a package in Ubuntu (not the first time I wanted to do this, but it was the first time I needed to do it for Ubuntu). So how to do it? Oh my, absolutely not docs, no options, no helpful threads, nothing. It seems Ubuntu people don't believe in the past. Talk about living with blinders on.

A good reference for the Apt tool is at https://help.ubuntu.com/community/AptGet/Howto

Now for some raw data.

You can find the most recent installation information in /var/log/apt/history.log which has entries like:

Start-Date: 2011-10-04  08:54:05
Upgrade: libwxgtk2.8-0:i386 (2.8.11.0-0ubuntu8, 2.8.11.0-0ubuntu8.1), libwxbase2.8-0:i386
(2.8.11.0-0ubuntu8, 2.8.11.0-0ubuntu8.1), python-wxversion:i386 (2.8.11.0-0ubuntu8,
2.8.11.0-0ubuntu8.1), rsyslog:i386 (4.6.4-2ubuntu4, 4.6.4-2ubuntu4.1), python-wxgtk2.8:i386
(2.8.11.0-0ubuntu8, 2.8.11.0-0ubuntu8.1)
End-Date: 2011-10-04  08:54:12

Cloning a machine's package list can be done using the commands:

sudo dpkg --get-selections > packages.txt
scp packages.txt remote-machine:./
sudo dpkg --set-selections < packages.txt
sudo apt-get dselect-upgrade

The following command just provides the list of packages:

sudo dpkg --get-selections 2>/dev/null | cut -f1

The following command provides the list of packages which where auto installed:

awk -v RS= '/\nAuto-Installed: *1/{print$2}' /var/lib/apt/extended_states

The following command provides the list of packages which where manually installed:

awk -v RS= '/\nAuto-Installed: *0/{print$2}' /var/lib/apt/extended_states

Discovering what versions of a package are available using the command:

$ apt-cache showpkg python-wxgtk2.8
Package: python-wxgtk2.8
Versions: 
2.8.11.0-0ubuntu8.1 (/var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_natty-updates_universe_binary-i386_Packages) (/var/lib/dpkg/status)
 Description Language: 
                 File: /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_natty-updates_universe_binary-i386_Packages
                  MD5: d5987e02966374dda23a807d4f9ed568

2.8.11.0-0ubuntu8 (/var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_natty_universe_binary-i386_Packages)
 Description Language: 
                 File: /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_natty_universe_binary-i386_Packages
                  MD5: d5987e02966374dda23a807d4f9ed568
[...]

To lock down a package so it doesn't get upgraded can be done by pinning it (a good reference for the various ways to do this is at https://help.ubuntu.com/community/PinningHowto). For the purposes here the command line version is:

echo "[package] hold" | sudo dpkg --set-selections

To unlock the packafe you mark it for installation like the following.

echo "[package] install" | sudo dpkg --set-selections

And finally to install a the specific version the command is:

sudo apt-get install [package name]=[version]

So for example to install the older version of python-wxgtk2.8 above and pin it the commands would be:

sudo apt-get install python-wxgtk2.8=2.8.11.0-0ubuntu8
echo "python-wxgtk2.8 hold" | sudo dpkg --set-selections

Now after all that research and poking I figured out the list of what packages might be causing me the issue (hint it wasn't actually python-wxgtk2.8 which I am using just cause it is a good example). Thus began the tedious task of installing earlier versions, testing to see if the issue went away, if not putting back the most current version. Finally I found the problem package, pinned it, added the command to the cron to unpin the package in the future, and made an appointment in my calendar to confirm the package unpinned as well as what I needed to do to cross-check everything.

$ sudo apt-get install openssl=0.9.8o-5ubuntu1
$ echo "openssl hold" | sudo dpkg --set-selections
$ sudo at 4am + 3 months
at> echo "openssl install" | dpkg --set-selections
at> ^D
$ at 8am + 3 months
at> mail -s "Remember to cross check unpinning of openssl" me@mydomain.com << EOF
at> Yo
at> 
at> Confirm python-wxgtk2.8 was unpinned on vm01:
at>  sudo dpkg --get-selections | grep openssl
at> 
at> Was pinned because caused error X1 for service Y1 via scripts Z1 and Z2.
at> 
at> Cheers
at> Yourself
at> EOF
at> ^D
$