Sunday, July 12, 2009

Fedora 11 - Upgrade from Fedora 10... not good... not good at all

Overview
So there I was, sitting at my workstation, doing a bit of work on my Fedora 10 server. When, all of a sudden a docile little dialog appeared notifying me that Fedora 11 was available and I could run pre-upgrade before upgrading.

Well, being an Ubuntu user, I have done many simple and smooth network based upgrades. So I figured, why not. Lets give it a shot. All my critical data is backed up anyway.

Now, I know what you are thinking. It states on the fedora project website that they prefer a clean install vs an upgrade. However, that doesn't mean an upgrade isn't possible? Right. I mean if your system pops a dialog saying "Distribution upgrades available" you should feel comfortable doing it. If it is possible but not recommended, why make the option so obvious.

I have been using Redhat/Fedora for as long as I have been using linux. Normally, I have followed the advice of doing a clean install vs upgrade. This time, I did the upgrade via network and let just say, it was not a good experience. Very little worked as advertised and I was always this close to system corruption and rebuild.

Now, I am not saying that an upgrade path from Fedora 10 to 11 spells doom. I am just saying that it almost did for me.

Background
My system has been upgraded twice by DVD. From Fedora 7 to 9. Then from Fedora 9 to 10. This may have set the stage for my upgrade distress. In all cases, there were some issues and bugs that I had to correct. Before FC7, I only did clean installs.

What Happened
My issues started during the pre-upgrade process. The updates dialog pops letting me know that FC11 is available. When clicked, the update process looks at your system and stages what it needs to upgrade.

During this phase, I got a message saying that the install image would not fit in /boot that that I needed a wired network after reboot to get the image during the upgrade. I had a wired network so I let it go.

After reboot, being sure to hit enter when you see the boot countdown (not documented) to get the option for the upgrade boot kernel.

Once I got into the actual upgrade process, I started getting error messages saying that it could not find packages (RPMs). That I could reboot or retry. The dialog said that reboot would mean possible corrption of my system. Retry, would simply retry (which is stupid). There was not a third option like, specify new location, data source, or continue anyway.

What is worse, is that it appeared that the installer was expecting RPMs to be in a different directory then they were in the install image (!!!). How this is possible, I have no idea since I never specified either. The update handled the mirror selection and all the behind the scenes work.

Well, corrupting my server, requiring a rebuild was just not going to happen. The following are a few thing I did to get through the mess and to a final working system.

Problem - Cannot Find Package to Install
It is possible that Fedora expected their install to barf so they were kind enough to provide most of the tools needed to correct them.

gives you a prompt. Here you will save the day.
gives you the current console log. Here you will see what is missing.

Two main tools that saved the day
  • wget
  • lynx
Both are text based and will let you fetch one or many missing files. lynx will complain since it needs a config. In /mnt/.../etc (look around) you will find a lynx.cfg and a lynx.lss. Copy them to /etc/. wget simply works.

Next thing you will need is a good working mirror. You will need to visit the fedoraproject site for that to find one. Each system will be different so you wil need to look at the console log to see where it is expecting to find the RPMs. Once you can see that (i.e. file:///mnt/system/...) you can cd to that directory and fetch each missing file one by one as you see the errors.

This job actually sucks since a missing gnome-libs problably means that gnome-libs-devel will be needed later. To get around this, I worked in broad strokes.
  1. I fetch a list of all the files using wget from the mirror site.
  2. I washed the "packages.html" file through cut and awk to get a list of only the rpm files
  3. This list of files was then washed through awk a second time to pre-pend wget http://..../file.rpm.
  4. This was then redirected into a text file for later use
When an error would pop up. For example gnome-libs. I would simple run the following from the prompt:

cat packages_wget.txt |grep gnome-libs |sh

Now the file (or any file with gnome-libs in it) will get fetched. This does not mean that it will get automatically installed but will be available for the installer.

I did this for all missing files. Later during the upgrade phase, there were more errors, so I followed the same approach.

I now what your are thinking
Why didn't I stop the install and download a DVD? I thought about that, however, the error messages dialog warned about corruption to the system and it was not clear how many files were actually missing. I might have been on my "last one". Once I got the above approach working, it went pretty fast. It was just an unforgiveable waste of time.

Hurray, the system upgrade is done. Reboot and Run Updates...!!!..Grumble
Finally, the upgrade completed. I rebooted the system and prepared for doing a yum update. This should be cut and dry now.... right? Nope. Aparently, there are several pre-release packages that were included in something the system was upgrading. Others had errors.

dhcp and cyrus-imap had a refernce to an openssl library that had an incorrect symlink. Looking at the error and correcting the symlink did the trick.

In a nutshell, yum update tried to upgrade a packages that does not exist on any mirror or even on the fedora site. Unless you know the full dependancy tree within the 631 updates, it is hard to turn off the ones in questions. For this reason, I had to upgrade in small batches until I found the packages that had the dependancies to the other packages that didn't exist.

Final Thoughts
I don't think I can stomach another Fedora upgrade. Will this happen to you? I am not sure. There could have been many reasons for my hardship, however, I have been using Fedora/Redhat for at least 10 years so I now the ins and outs. Install is not a strong point. It is flexible and easy to manage once running however. It is my (current) choice for a server OS.

My thought to you is: if the installer tells you that you do not have room on your /boot partition, cancel and walk away. I have a feeling that a DVD install would have gone better. A clean install would be best.

I just wish they could make it work like Ubuntu. That distro is great for package management. They just tend to be slower on updates (i.e. Java).

I will try to add more details about specific commands and syntax. My main goal here was to put information out, in the event, that other of you have run into the same issues.

Update 2009-Sept
Since my painful upgrade, the updates have been flowing just fine. My system has been running pretty well.