Things n' Stuff Thoughts about the universe in general

Its time to update an old post about updates: Fedora 7 is again rather old, with the current Fedora being at 12, and 13 is right around the corner, but We’re still with CentOS 5 (I hear that RedHat is deep into beta with their next version – 6 but when it comes out is anyone’s guess).

In the mean time, if you want to run something a bit more modern maybe you’d want to upgrade your CentOS installation to Fedora 12 which can be considered pretty stable after a few months on the market. The reasons for the change may be numerous – maybe you need to run the latest subversion (with better merge tracking) or a new version of Ruby. Either way, if upgrading from CentOS 5 to Fedora 7 was a pain, guess how it is more the two years later?

Before we get to the action itself, I would be remiss if I would not recommend you to reconsider what you are about to do – this is a challenging exercise for many system admins and can sometimes end badly. I do not guarantee anything and if it breaks you get to keep both parts. Often its much faster and easier to just go to the server, stick a Fedora 12 DVD in the slot, reboot and let the DVD upgrade your system¹. That being said, this is fun if you’re into that sort of thing, so if you’re still with me, lets dive in!

So Bring On the Pain

We’ll start with a fully updated CentOS 5.4 – there’s no reason for getting more pain then necessary, and upgrading a CentOS 5.3 or earlier to 5.4 is painless – just run yum upgrade -y and grab yourself a cup of coffee (notice that we use “upgrade” and not “update” – theoretically this may result in old packages being removed, so pay attention, but from my experience EPEL based distributions are very stable and do not normally get stuff obsoleted during a major version’s lifetime). Upgrading will also usually install a new kernel, so if that happens please do a reboot before continuing. Important: This is also a good time to disable SELinux if you have it in “enforcing” mode – after the upgrade the security labels will be in a mess and with SELinux enabled its going to break your system and you will not be able to log back in. To disable it, edit /etc/sysconfig/selinux and in the line that says SELINUX=enforcing change the value to “disabled”, save and reboot the machine – SELinux mode is set only during boot time.

Its also important to note that this procedure is based on a default install of CentOS 5.4 – while its likely that it will work without changes on a machine upgraded from a previous CentOS, if you customized your installation then some packages that I assume are installed, notable a GNOME desktop, may be missing and some procedures listed below will break unexpectedly – if this happen use your best judgment or contact me for advice.

The next thing we need to know before installing, is where to get Fedora 12 packages from: You can find these at a mirror close to you – look for a mirror in the Fedora 12 mirror list. After you select your mirror, go into releases/12/Fedora² where you will find 3 directories – under source/SRPMS you’ll find your source RPM packages (which you will need a few); under i386/os/Packages or x86_64/os/Packages (depending on what architecture you currently have installed) you’ll find the relevant binary packages.

RPM Incompatibilities

The main problem we have to address when upgrading to Fedora 12 is that the RPM format has changed with RPM version 4.6, and the version of RPM installed in CentOS 5 (version 4.4) can’t be used to install packages from Fedora 12. So we’ll need to start by building a new version of RPM from source and install them. Start by installing a some required packages using the command
yum install -y mc rpm-build redhat-rpm-config elfutils-devel elfutils-libelf-devel readline-devel zlib-devel nss-devel gettext-devel libselinux-devel ncurses-devel bzip2-devel python-devel libcap-devel libacl-devel libtool gcc-java gcc-c++ java-1.6.0-openjdk-devel audit-libs-devel pam-devel texinfo sharutils doxygen

We’ll use mc to extract the source files from the source RPM files that are packed in a new format our current RPM installation does not understand, and rpm-build to build those sources. The rest of the files are development libraries that are needed by software we will be building.

If you are doing this on a 64bit CentOS install, you should make sure to remove the 32bit versions of packages you have installed – CentOS, like all Fedora based operating systems, install a lot of 32 bit packages by default for backward compatibility, and these will cause a lot of conflicts during build and installation, so better remove them before starting. To remove them, the easiest way is to run the command
yum remove glibc.i686

This will remove a lot of packages and all but a few should be 32 bit packages (the few 64bit packages that are removed should be safe to remove). If you actually need 32bit support, you can reinstall it later.

Building Required Software

The next thing we need to do is get some source RPM packages from Fedora 12 source mirror, extract them, build new binary packages that work on our system and install the new binary packages. This process boils down to the following steps that have to be repeated for each package:

1. Download the src.rpm file: from the Fedora mirror, from the source/SRPMS directory we discussed previously. I usually cd /usr/src/redhat and download there – it saves some cding back and forth and still keeps things relatively organized.
2. Extract the source code: we need to use Midnight Commander to extract the source code because the RPM version installed in CentOS does not understand the package format of the new RPM files from Fedora 12 – if you try to install the new RPMs directly you’d get a lot of checksum errors. So start mc in the directory where you downloaded the package. From the MC view, locate the src.rpm file you just downloaded, position the cursor over it using the arrow keys and press ENTER. you should now be “inside” the source package and should see a pseudo-file named CONTENTS.cpio. Position the cursor over the content pseudo-file and hit ENTER again to get to the actual archive content. Now hit TAB to move to the second file view and navigate to /usr/src/redhat/SOURCES – if you followed my advice in the previous step, it should be right in the same directory. use TAB to get back to the source archive, mark all the files by pressing * and press F5 and ENTER to copy them to the sources directory. Now exit MC by using the F10 key.
3. Build the packages: navigate to /usr/src/redhat/SOURCES and there should be a file named <package-name.spec>. This is the RPM specification file with the instructions on how to build the required binary packages. Use the command
rpmbuild -bb --clean --rmsource --rmspec <package-name&gtl.spec
to start the build. Sometimes you will also need to add the switch --nodeps if the RPM specification lists some requirements which we do not want to install and are not really mandatory to build. This is not safe to do per-se, but we will use this when we have to. If the build completes successfully, then the output from the build should end with something like this:
+ exit 0
. If the build was completed successfully, then rpmbuild will also have removed the source files automatically as they are not needed anymore.
4. Installing the new software: once the build process completes, it should note that it “wrote” some binary RPM files. Install all these files using the command
rpm -Uhv <list of binary RPM files>
with the full paths of all the RPM files the build process listed. If there are RPM files which end in “debuginfo” or “static”, you don’t need to install these, but do install “devel” packages. Sometimes I’ll ask you to also add the switch --nodeps to this command as well, or even --replacefiles. Again this is not safe per-se, but a bit of ambiguity here can save a lot of time.

So now that you know what to do, lets get to work:

1. Download tcl-8.5, build it with --nodeps and install it with --nodeps --replacefiles
2. Download db4-4.7, build it with --nodeps and install it using rpm -ihv instead of rpm -Uhv and also add --nodeps
3. Download popt-1.13, build it with --nodeps and install it using --replacefiles
4. Download file-5.03, build it normally and install it using --nodeps
5. Download libcap-2.16, build it normally and install it using rpm -ihv instead of rpm -Uhv and also with --replacefiles
6. Download lua-5.1.4, build it normally and install it normally
7. Download fakechroot-2.9, build it with --nodeps and install it normally. I’ve seen that some mirrors are missing this package for some reason, if your mirror does not have it, try other mirrors or try the ISOC Israel mirror where I found this package: http://mirror.isoc.org.il/pub/fedora/releases/12/Everything/source/SRPMS/
8. Download xz-4.999 from my RPM site at http://rpms.geek.co.il/centos5/SRPMS³, build it normally and install it normally. We can’t simply build it from Fedora source repository because the Fedora source package for xz uses the XZ compression format – so you need xz installed in order to build and install xz, kind of a chicken and egg problem. Weird.
9. Download rpm-4.7, build it normally and install it with --nodeps

Setting Up

Now that we have a new RPM that can install Fedora 12 packages, its time to get the Fedora release package – this contain the software repository configuration that we need to upgrade our system to Fedora 12. You can download these files from the mirror directory for the architecture you need – look for the package fedora-release-12-1.noarch.rpm. But before we install it, we need to remove the old centos-release package as it will conflict with the new package – so run
rpm -e centos-release --nodeps

then run

rpm -Uhv fedora-release-12-1.noarch.rpm


Its All Downhill From Here

Now we’re just left of installing the new system. Its not as easy as doing a yum upgrade and its a bit more complicated then the Fedora 7 upgrade we did 2 years ago, but its manageable.

We’ll start by removing some development packages and other not important packages that can cause us to waste time on conflict resolution. Run the command
rpm -e $(rpm -qa '*-devel') gcc-java elfutils-libelf-devel-static elfutils-devel-static gcc gcc-c++ rpm-cron rpm-build rpm-apidocs glibc-headers pilot-link gnome-pilot evolution evolution-connector launchmail


Now we need to download some packages and install them manually before yum can work properly. From the Fedora 12 mirror of your architecture (32bit or 64bit), download these packages:

audit
audit-libs
audit-libs-python
binutils
ca-certificates
curl
dirmngr
e2fsprogs
e2fsprogs-libs
glib2
glibc
glibc-common
gnupg2
gpgme
libcom_err
libcurl
libffi
libksba
libss
libssh2
libuser
lua
ncurses
ncurses-base
ncurses-libs
nscd
nspr
nss
nss-util
nss-softokn
nss-softokn-freebl
openldap
openssl
pam
passwd
pinentry
prelink
pth
pygpgme
python
python-libs
python-pycurl
python-urlgrabber
readline
rpm
rpm-libs
rpm-python
shadow-utils
sqlite
util-linux-ng
wget
yum
yum-metadata-parser


Then install all the downloaded packages at the same time by using the command
rpm -Uhv --nodeps
followed by all the files that you downloaded. I normally create a directory, download all the files there, and then use rpm -Uhv --nodeps *. If all went well so far, then you should see that RPM is installing all the files without problems (there may be a few configuration files that are saved, but that’s OK). At the end of the installation, you may get some errors about triggers failing because of missing libtermcap.so.2 – that is expected and you can ignore it for now.

At this point running YUM should work with the new version – try it out by invoking yum clean all, it should complete without errors. If you have an error in this stage then I suggest going back and checking what you missed.

Just About Done

All that we have left to do now, is upgrade the system using the new YUM to upgrade all the packages to the latest versions from the Fedora 12 repository. The first thing is to upgrade BASH, because the current (old) version of BASH we have installed is broken and unless we fix it, all the RPM scripts will break. So run the command
yum install -y bash


We’ll need to resolve a few other problems before we can run the full upgrade:

• The GCC Java package installed in CentOS 5.4 does not upgrade well to the newer GCC Java, so we’ll need to remove it by hand – run the command
rpm -e java-1.4.2-gcj-compat --nodeps
to get rid of it. The new GCC Java package will be installed automatically as part of the upgrade.
• The Fedora logos package can’t automatically replace the CentOS provided one, for some reason, so we’ll need to do it by hand:
yum install -y fedora-logos

• The DeviceKit package conflicts with the old kernel, but YUM dependency resolution fails to take into account that we are installing a new one, so we’ll need to upgrade the kernel manually. There is also another conflict of very old packages with very new ones that we’ll need to resolve, so run
rpm -e rhgb mkinitrd --nodeps
yum upgrade -y kernel
rpm -e $(rpm -qa kernel* | grep 2.6.18)


Before we do the full upgrade, lets make sure that we can log in after the system reboots. We’ll update OpenSSH and some required files:
yum upgrade -y openssh-server 'pam*' '*selinux*'
(the quotes are important). Now restart the SSH server:
service sshd restart

Now try to log in to the server, which should work (if you disabled SELinux at the start).

Now, finally, we can run the full upgrade. Execute
yum upgrade -y

and go do something more fun for a while – it will take some time to download and install more then 1000 packages.

After that is done, we need to recreate the initial ramdisk for the boot process to complete, because the one created during the upgrade is broken. Run this command:
dracut -f /boot/initramfs-2.6.*.img $(rpm -q --qf '%{version}-%{release}.%{arch}' kernel)


Lastly, we want to make sure that the system is using the new style boot process using upstart and not using the old inittab file to manage the console, otherwise when we reboot we won’t have a local login console. So run
mv -f /etc/inittab.rpmnew /etc/inittab


When the dust settles, you should be able to reboot to a working Fedora 12 installation.

It is also recommended to look in the file system and find all the backups of configuration files in the form *.rpmnew files and *.rpmsave file and merge any changes in them back to the original configuration files. rpmnew files are new configuration files that the upgrade has brought in and contain new functionality that should be put in – instead of forcing the change they are backed up with a .rpmnew extension. rpmsave files are configuration files that had local changes but should have been removed – instead of losing the changes, they are backed up with a .rpmsave extension.

What I normally do is run
find / -name '*.rpmsave'
and go over each file and decide if it has important configuration and if so figure out where it should go now. Then I delete the rpmsave file. I then run
find / -name '*.rpmnew'
and go over each file by comparing it to the current configuration file using a diff tool⁴ and put in the new configuration that you like. If you know for a fact that you do not care about the specific configuration file, you can just move the rpmnew file over the old configuration.

Good luck, and let me know if you have any issues with this procedure. Again – there are so many pits here that an unwary traveler may fall into. I tested this procedure on a VM and had to restore it from snapshot more then once, and its important to note that if you get disconnected from the machine during the process, you won’t be able to restore it as after about the second step the SSH server will not accept new connections until the full upgrade is done. I believe that by following this procedure, and maybe with a bit of creative thinking, one can successfully upgrade a running CentOS 5 server, but it won’t be completely without downtime and there is a lot of risk in this. If the service is very important to you – go to the server farm and stick a disc in there :-).

1. another safer options would be to upgrade to Fedora 7 using the instruction in the previous article linked above, and then use Fedora’s preupgrade tool to upgrade to Fedora 8, then to fedora 10, and from there to Fedora 12, though this procedure will also require physical access to the machine and multiple reboots and multiple long downloads [↩]
2. Its important to go to the original release version and not be tempted to download packages from the “updates” repository, as during the upgrade procedure we are going to install some packages badly, on purpose, but the update at the end will fix them. [↩]
3. alternatively you can get the 32bit or 64bit packages directly from the respective directory under http://rpms.geek.co.il/centos5/ [↩]
4. I like meld, but it requires an X server on the machine you are working from, so unless you do this locally or from a remote Linux machine it will probably not work for you. Alternatively you can either use vimdiff or a simple text mode diff [↩]

This entry was posted on Friday, February 12th, 2010 at 06:32 and is filed under Articles, Software. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.