Perseus Software Configuration Changes in 2016

Links below were the guides to what was done. It should be assumed that the indicated steps were executed by root.

Major system update, in 2016... which unfortunately ran into major difficulties!

After one full year, the time seemed ripe for an update from CentOS 7.0 to 7.2, plus a complete software refresh.
It was hoped this would address several unexplained incidents where Perseus became entirely unreachable (even via the console).
In principle, this simply should have been a matter of running "yum update", waiting for an hour or so, and rebooting...
Instead, the server hit one of its "unreachable moments" while the update was only partway done(!), leaving a total mess.

After a cold boot, the GNOME desktop at the console was broken, presumably because some of its pieces were updated and some were not.
Solution: "Ctrl + Alt + F4" brings up a command line (F1-F7 may also work) - [SOLVED] "Oh no! Something has gone wrong." on Restart

This began a day-long recovery process... documenting it in full is impossible... only the most helpful commands are recorded below.

The first move was to try "yum-complete-transaction", but this seemed to get into an infinite loop and had to be interrupted (again!).
Therefore, some additional cleanup steps were attempted, in the hope of reaching a state where "yum update" could be re-run.
Difference between yum update vs yum install
[SOLVED] graceful recovery from interrupted yum update
Fixing There are unfinished transactions remaining...

yum-complete-transaction
yum clean all
yum makecache
rpm --rebuilddb
yum update

The repeat of "yum update" went nowhere due to dependency issues involving multiple packages. What to do in such a case?
Unresolved dependency after yum update crash (broken ssh connection)
yum update or yum install fails with package conflict...
Yum file conflicts due to my mistake. How do I fix?
[SOLVED] Yum Transaction Check Error ["file X from install of Y conflicts with file from package Z"]
Found 3 pre-existing rpmdb problem and no fix
yum - Package Conflicts

In most cases, the conflict arose because two versions of the same package were installed.
Simply doing "yum reinstall" turns out to be not very useful, as yum just detects the most recent package, and it does nothing.
That's probably why the above links recommend a general strategy of just removing selected packages and installing them over.
But: "yum remove" also removes all packages that depend on the selected one--and there is no easy way to re-install the whole bunch.

Thus the trick is to focus on packages with few or no dependencies, and for each, re-install the entire list following the "yum remove".
(This seems preferable to using commands that remove the package while ignoring dependencies, e.g., "rpm -e --nodeps".)
Problem packages were identified with "package-cleanup --problems". These were dealt with one at a time until none remained.

package-cleanup --problems
yum remove ...
yum install ...

(Much of this work was done via ssh, after ssh was updated separately and confirmed to be back in working order.)
When the outstanding issues were fixed, "yum update" was able to proceed to completion!!
However, yum only updates packages that are present; yum skips packages that were removed, unless they were installed again.
How can one tell if all the pieces are really back in place?

The largest dependency lists were associated with Linux desktops--is there a way to be sure all these packages were truly restored?
One possible answer is to do a "yum groups install", which checks to ensure all essential packages for a given desktop are present.
Linux YUM (Yellowdog Updater, Modified) Commands for Package Management
How to install a group of packages with yum on Red Hat Enterprise Linux?
Yum groups and repositories

yum groups install "GNOME Desktop"
yum groups install "MATE Desktop"

Another possible answer is to try "yum check"--but that command turned out to be suspiciously slow, and the wait didn't seem worth it.
So, what do I do with the output of yum check?
`yum check` enters endless loop

The only sure-fire way to make certain that no packages have been omitted is to examine yum's history for every "erase" command.
Each package that was erased needs to be reinstalled (not just updated) to make sure it is not missing from the final collection.
CentOS / RHEL: See Detailed History Of yum Commands
YUM History (list, info, summary, repeat, redo, undo, new)
CentOS: Install packages listed in a text file? [e.g., a list of packages that were erased according to yum history info #]

yum history all
yum history list 1..85
yum history info 13
yum install ...

The process was fairly tedious. but it caught quite a few oversights, while eliminating a lot of uncertainty about the recovery.

To make sure all the old packages were really gone and to verify a clean final state, the following commands were run:

package-cleanup --cleandupes
package-cleanup --dupes
package-cleanup --problems

Finally, the kernel was reinstalled as an extra security measure prior to rebooting. The reboot was successful!
How To Update the Kernel in CentOS / Red Hat
How is Grub updated after each kernel upgrade?

New packages added on 4/22/16: SciPy, SymPy, and python-matplotlib were added via "yum install" to improve Python functionality.

System refresh on 5/5/16: All DIMMs were reseated; all packages (but the kernel) got an update with "yum update --exclude=kernel* ".



Last updated on 5/1/17 by Steve Lantz (steve.lantz ~at~ cornell.edu)