In Ubuntu there are a number of ways to get upgrades. There is the update manager which is configured to run automatically. There is using the command line approach with apt-get ('sudo apt-get update && sudo apt-get upgrade'). Finally there is the GUI utility called Synaptic ('sudo synaptic').
So what happens when something gets updated and introduces a bug, or breaks another applications behavior?
I had this happen to me just this week. I was running along great, my services were all happy. Then the update manager popped up saying there were some new upgrades. Being security minded I like to keep all my libraries at the most current level. Of course I looked over the list and there was an update to the ssl libraries. Nothing concerning in the release notes for the changes. So I let it get installed.
Some time later I finished changing some REST services, had a clean build and deployed to my testing Tomcat server. At which point my integration tests failed. Poking at the problem and turning up the verbosity got the error message:
SSLError: [Errno 8] _ssl.c:503: EOF occurred in violation of protocol
That was new. I hadn't changed the Python script that was producing this error message. Naturally I thought it was all the changes I had made. I Googled and Googled and read and found nothing even remotely helpful. OK so I then checked against the production services. OH MY they were producing the same error! Panic adrenalin is sometimes good sometimes not. In this case not. I spun and started calling other developers. They all checked and hey no issues. Hmm OK no issues, really? Alright then it must just be me. Wait a tick didn't I install some patch to SSL. Oh ya. OK check another machine that is not running that patch. Sure enough everything works great.
You'd think problem solved at this point. But, you'd be wrong. (Or maybe not, you are after all reading this, which means it is likely you are having a similar problem.)
Thus began the quest to downgrade a package in Ubuntu (not the first time I wanted to do this, but it was the first time I needed to do it for Ubuntu). So how to do it? Oh my, absolutely not docs, no options, no helpful threads, nothing. It seems Ubuntu people don't believe in the past. Talk about living with blinders on.
A good reference for the Apt tool is at https://help.ubuntu.com/community/AptGet/Howto
Now for some raw data.
You can find the most recent installation information in /var/log/apt/history.log which has entries like:
Start-Date: 2011-10-04 08:54:05 Upgrade: libwxgtk2.8-0:i386 (2.8.11.0-0ubuntu8, 2.8.11.0-0ubuntu8.1), libwxbase2.8-0:i386 (2.8.11.0-0ubuntu8, 2.8.11.0-0ubuntu8.1), python-wxversion:i386 (2.8.11.0-0ubuntu8, 2.8.11.0-0ubuntu8.1), rsyslog:i386 (4.6.4-2ubuntu4, 4.6.4-2ubuntu4.1), python-wxgtk2.8:i386 (2.8.11.0-0ubuntu8, 2.8.11.0-0ubuntu8.1) End-Date: 2011-10-04 08:54:12
Cloning a machine's package list can be done using the commands:
sudo dpkg --get-selections > packages.txt scp packages.txt remote-machine:./ sudo dpkg --set-selections < packages.txt sudo apt-get dselect-upgrade
The following command just provides the list of packages:
sudo dpkg --get-selections 2>/dev/null | cut -f1
The following command provides the list of packages which where auto installed:
awk -v RS= '/\nAuto-Installed: *1/{print$2}' /var/lib/apt/extended_states
The following command provides the list of packages which where manually installed:
awk -v RS= '/\nAuto-Installed: *0/{print$2}' /var/lib/apt/extended_states
Discovering what versions of a package are available using the command:
$ apt-cache showpkg python-wxgtk2.8 Package: python-wxgtk2.8 Versions: 2.8.11.0-0ubuntu8.1 (/var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_natty-updates_universe_binary-i386_Packages) (/var/lib/dpkg/status) Description Language: File: /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_natty-updates_universe_binary-i386_Packages MD5: d5987e02966374dda23a807d4f9ed568 2.8.11.0-0ubuntu8 (/var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_natty_universe_binary-i386_Packages) Description Language: File: /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_natty_universe_binary-i386_Packages MD5: d5987e02966374dda23a807d4f9ed568 [...]
To lock down a package so it doesn't get upgraded can be done by pinning it (a good reference for the various ways to do this is at https://help.ubuntu.com/community/PinningHowto). For the purposes here the command line version is:
echo "[package] hold" | sudo dpkg --set-selections
To unlock the packafe you mark it for installation like the following.
echo "[package] install" | sudo dpkg --set-selections
And finally to install a the specific version the command is:
sudo apt-get install [package name]=[version]
So for example to install the older version of python-wxgtk2.8 above and pin it the commands would be:
sudo apt-get install python-wxgtk2.8=2.8.11.0-0ubuntu8 echo "python-wxgtk2.8 hold" | sudo dpkg --set-selections
Now after all that research and poking I figured out the list of what packages might be causing me the issue (hint it wasn't actually python-wxgtk2.8 which I am using just cause it is a good example). Thus began the tedious task of installing earlier versions, testing to see if the issue went away, if not putting back the most current version. Finally I found the problem package, pinned it, added the command to the cron to unpin the package in the future, and made an appointment in my calendar to confirm the package unpinned as well as what I needed to do to cross-check everything.
$ sudo apt-get install openssl=0.9.8o-5ubuntu1 $ echo "openssl hold" | sudo dpkg --set-selections $ sudo at 4am + 3 months at> echo "openssl install" | dpkg --set-selections at> ^D $ at 8am + 3 months at> mail -s "Remember to cross check unpinning of openssl" me@mydomain.com << EOF at> Yo at> at> Confirm python-wxgtk2.8 was unpinned on vm01: at> sudo dpkg --get-selections | grep openssl at> at> Was pinned because caused error X1 for service Y1 via scripts Z1 and Z2. at> at> Cheers at> Yourself at> EOF at> ^D $