Server Upgrade

From SEDSWiki

Jump to: navigation, search

Contents

General Overview

  1. Perform Base Configuration of New Server
  2. Prepare old server for migration
  3. Copy data to new server
  4. Configure Services on New Server
  5. Take old Server Offline
  6. Perform Final Synchronization
  7. Update DNS Records to make new server Prime
  8. Rebuild old Server as Development Platform

Opportunities with New Server

There are a few things to consider when moving to the new server - opportunities to stay with existing software and approaches, or try out new things. The following are choices under consideration in moving to the new server:

Change from Fink to MacPorts

The old server made use of Fink for various pre-compiled and pre-configured UNIX utilities and libraries. In moving to the new server, we'd like to change over to using Mac Ports for the same purpose. It appears to have a wider repository, tighter control over installation options, etc, etc.

In performing this cross-grade, we'll want to make sure that all the utilities that we have installed in Fink are installed in MacPorts.

The list of currently installed software in Fink can be retrieved as follows: 

fink list --installed

Since we don't need any of the "virtual" packages to be re-created, those can be filtered out as follows: 

fink list --installed | grep -iv virtual

apt    0.5.4-52        Advanced front-end for dpkg
apt-shlibs     0.5.4-52        Advanced front-end for dpkg
autoconf2.5    2.59-6  System for generating configure scripts
automake1.9    1.9.4-1 Tool for generating GNU Standards-compliant Makefiles
base-files     1.9.6-1 Directory infrastructure
bison  1.35-2  Parser generator
bzip2  1.0.2-12        Block-sorting file compressor
bzip2-dev      1.0.2-12        Developer files for bzip2 package
bzip2-shlibs   1.0.2-12        Shared libraries for bzip2 package
cctools-extra  1:525-1 Extra software from cctools
debianutils    1.23-11 Misc. utilities specific to Debian (and Fink)
dlcompat       20030629-15     Dynamic loading compatibility library (dummy)
dlcompat-shlibs        20030629-15     Dynamic loading compatibility library (shared libs only)
dpkg   1.10.21-217     The Debian package manager
fileutils      4.1-4   Common shell commands like ls, touch, chmod
fink   0.23.10-11      The Fink package manager
fink-mirrors   0.24.4.1-1      Mirror infrastructure
fink-prebinding        0.7.1-2 Tools for enabling prebinding in Fink
gd2    2.0.33-2        Graphics generation library
gd2-shlibs     2.0.33-2        Shared libraries for gd2 package
gdbm-shlibs    1.8.0-7 Shared libraries for gdbm package
gettext        0.10.40-19      Message localization support
gettext-bin    0.10.40-19      Executables for gettext package
gettext-dev    0.10.40-19      Developer files for gettext package
gettext-tools  0.10.40-19      Developer executables for gettext package
ghostscript    8.00-3  Interpreter for PostScript and PDF
ghostscript-fonts      6.0-3   Standard fonts for Ghostscript
gzip   1.2.4a-6        The gzip file compressor
ispell 3.2.06-3        Interactive spell-checker
libiconv       1.9.1-11        Character set conversion library
libiconv-bin   1.9.1-11        Executables for libiconv package
libiconv-dev   1.9.1-11        Developer files for libiconv package
libjpeg        6b-16   JPEG image format handling library
libjpeg-bin    6b-16   Executables for libjpeg package
libjpeg-shlibs 6b-16   Shared libraries for libjpeg package
libncurses5-shlibs     5.4-20041023-6  Shared libraries for libncurses5 package
libtiff        3.6.1-4 TIFF image format library and tools
libtiff-bin    3.6.1-4 Executables for libtiff package
libtiff-shlibs 3.6.1-4 Shared libraries for libtiff package
links-ssl      0.98-12 Lynx-like text WWW browser with tables
m4     1.4.2-1 Advanced macro processing language
ncftp  3.1.7-1 Browser program using ftp protocol
ncurses        5.4-20041023-6  Executable files for ncurses
ncurses-dev    5.3-20031018-501        Development files for ncurses package
ncurses-shlibs 5.3-20031018-501        Shared libraries for ncurses package
openssl097     0.9.7d-1        Secure Sockets Layer and general crypto library
openssl097-shlibs      0.9.7d-1        Secure Sockets Layer and general crypto library
pine   4.58-21 Text based tool for managing emails
pkgconfig      0.15.0-2        Manager for library compile/link flags
tar    1.14-1  GNU tar - tape archiver
texinfo        4.7-11  Texinfo documentation system
tidy   20021210-2      Utility to tidy up HTML code
type1inst      0.6.1-3 Type 1 PostScript font installation utility
unrar  3.3.5-21        RAR archive decoder
unzip  5.50-14 Decompression compatible with pkunzip
wget   1.8.2-2 Automatic web site retreiver

For starters, we'll probably run 'port install php5 +apache2 +macosx +mysql5 +pear +pspell +t1lib +tidy'

Move to DSPAM as sole Spam Filter

The current setup of Amavis+SpamAssassin+Dspam+ClamAV is effective, but takes a lot of time by Chris to keep trained, running smoothly, up to date, and often takes the whole system down when it isn't working perfectly. While there are a few other options out there, the best immediate path forward seems to be to switch to Dspam only, and let the users do their own training and correction.

This will require a few things to be setup on the new server: (Chris will do all these)

  1. Fresh setup of DSPAM set up in its new primary role
  2. Existing dspam mysql database to use as the corpus for starting the training for others
  3. Web server setup to allow access to DSPAM web utility through user authentication

We'll need to retain the "seds" account to serve as the gateway into the eSupport system, and now SEDS Exec board members can do their own training on it...

Change Filesystem Backup Strategy to Apple's TimeMachine

Chris has been using Apple's Time Machine for a while on his home 10.5 Server Setup, and its saved the bacon a few times. It requires some small modifications to the config file to backup all the server files, but it is better than rsync backup for the following reasons:

  1. Backups also use file-hard-linking for unchanged files, but Apple has modified their file system device drivers so they can also do directory hard-links (rsync can't do this), which can make incremental backups for large unchanged structures very quick.
  2. Which files have changed is also built into the kernel, so it backups are speedy
  3. Many more options for restoring files varying from user-level access, to catastrophic recovery of an entire failed system

Chris's notes on the particulars to edit: 

Location of files excluded in time machine backup:
/System/Library/CoreServices/backupd.bundle/Contents/Resources/StdExclusions.plist

Strings to comment out to ensure complete backups:
<!--		<string>/private/var/spool</string> -->

Backup frequency can be controlled in 
/System/Library/LaunchDaemons/com.apple.backupd-auto.plist

We'll still leave in the cron scripts for mysql backup and open directory backup.

Lights out Management

This is a nifty feature of the Intel Xserves that allows remote control of power and reset, regardless of what the OS is doing (good for rescuing a hung-system).

More about it here.

Perform Base Configuration of New Server

We'd like a clean starting point for building up the new server. First thing is to boot up off the Mac OS X Server 10.5 discs, format, and install the operating system. In the process of doing this, we'll want to set the machine up as an open directory slave of the old server, and before we're done, we'll promote it to an open directory master.

Tasks:

  1. Find and retrieve the Mac OS X Server system disks (Probably from Kenny Fine, Tony Ferro or Joe Gotobed)
  2. Figure out what the IP address of the server is (newseds.lpl.arizona.edu = 128.196.60.21)
  3. Get the DVD physically into the server, and start it from the disc - details on how to read and use the indicator lights to select CD startup are on page 12 of the user manual
  4. Find a Mac somewhere and run the Server Assistant application to walk through the installation process (alternatively, you could connect a USB keyboard and monitor, but I'm not sure if these servers have video cards)

FYI, when looking for material related to the new server, through Apple's wonderful model numbering system, it is known as the Xserve (Early 2008)

Useful references:

Perform Additional Configuration of New Server

Get packages from MacPorts installed. Here's what was on the old system. 

 apache2 @2.2.9_1 (active)
 apr @1.3.3_0 (active)
 apr-util @1.3.4_0+mysql5 (active)
 aspell @0.60.6_1+macosx (active)
 autoconf @2.62_0 (active)
 automake @1.10.1_0 (active)
 bzip2 @1.0.5_1 (active)
 curl @7.19.0_0 (active)
 cyrus-sasl2 @2.1.21_0+kerberos (active)
 db46 @4.6.21_1 (active)
 expat @2.0.1_0 (active)
 fontconfig @2.6.0_0+macosx (active)
 freetype @2.3.7_1 (active)
 gawk @3.1.6_0 (active)
 gd2 @2.0.35_1 (active)
 gettext @0.17_3 (active)
 gmake @3.81_0 (active)
 gperf @3.0.3_0 (active)
 help2man @1.36.4_1 (active)
 jpeg @6b_2 (active)
 libiconv @1.12_0+darwin_8 (active)
 libmcrypt @2.5.8_0 (active)
 libpng @1.2.32_0 (active)
 libtool @1.5.26_0 (active)
 libxml2 @2.7.1_0 (active)
 libxslt @1.1.23_1+darwin_8 (active)
 lzmautils @4.32.6_0 (active)
 m4 @1.4.11_0 (active)
 mhash @0.9.9_0 (active)
 mod_jk @1.2.25_0+jni (active)
 mysql5 @5.0.67_0 (active)
 ncurses @5.6_0 (active)
 ncursesw @5.6_1 (active)
 neon @0.28.3_0 (active)
 openssl @0.9.8i_0 (active)
 p5-locale-gettext @1.05_0 (active)
 pcre @7.8_0 (active)
 perl5.8 @5.8.8_3+darwin_8 (active)
 php5 @5.2.6_1+apache2+macosx+mysql5+pear+pspell+t1lib+tidy (active)
 pkgconfig @0.23_0 (active)
 popt @1.13_0 (active)
 readline @5.2.012_1 (active)
 rsnapshot @1.3.1_0 (active)
 rsync @3.0.4_0 (active)
 serf @0.2.0_0 (active)
 sqlite3 @3.6.3_0 (active)
 subversion @1.5.2_1+mod_dav_svn (active)
 t1lib @5.1.2_0 (active)
 texinfo @4.12_0 (active)
 tidy @20051026_0 (active)
 tiff @3.8.2_2+macosx (active)
 zlib @1.2.3_1 (active)

Prepare old server for migration

Cleaning up Log Files

Most of the logs of interest are in /var/log and /web/SEDSLogs.

Before we move, we should "clean up" the logs in /var/log and move the appropriate ones over to /web/SEDSLogs

MySQL Databases

It is my understanding that we are moving from MySQL version 5.0.67 on the old server to the same version of MySQL on the new server. With that assumption along with knowing that the hardware architectures are different I'll do my best to lay out the process of properly configuring the new server, transferring all 41 databases from the old server to the new server, and verifying that the transfer was successful. At no time will data on the old server be changed or dropped during this process.

According to the MySQL 5.0 reference manual section 2.18.5 Copying MySQL Databases to Another Machine transferring databases between differing architectures should be done using mysqldump as opposed to a straight binary transfer.

Since the mysql versions are the same the configuration should be easy, we'll just copy /etc/my.cnf to the appropriate location on the new server probably /etc/my.cnf and restart mysqld.

To generate the mysqldumps for the transfer we will use the existing backup script /backups/mysql/automysqlbackup.sh (it will be a good test of the backup script too). Access to the database should be stopped before running the script.

Special consideration needs to be taken for all databases that use innodb, triggers, or procedures (see /backups/mysql/_INNODB_restore_notes.txt). Essentially you need to make sure foreign key checks are turned off when loading data. The following databases use innodb: TBD. The only databases using triggers and procedures are protoforge and prototest, execute the sql script at /backups/mysql/protoforge_trigs_and_procs.sql after loading the data.

(under construction)

Copy data to new server

We'll use rsync to take care of the file moving between the servers. With the use of an appropriate script and include/exclude file, we can use the same script to perform the "bulk" copy as well as do the incremental copy of new changes just before we go live.

The basic command line arguments to be sure to include are: 

rsync -avE

In addition, you can do a dry-run with rsync using the n option. 

rsync -avE [[user@]host:]dir/ [[user@]host:]dir

It is important to remember how the trailing slash is handled with rsync: with the trailing slash in the from, it means copy all the stuff in the directory, not the directory itself, into the 'to' directory, which is usually what you want.

In general, most of the unique data on SEDS is in the following areas: 

/web
/Users
/migration (from the last time, going from Solaris to Mac)
/usr/local (src mainly, rest will/should be rebuilt)
/opt/local (most should be rebuilt from scratch)
/ftp - Ancient archive of FTP site (still in service!)

Mail Spool Files:
/var/imap
/var/spool/imap

/var/dspam (spam processor data)
/private/var/root (root home directory)
/var/cron/tabs (cron tabs)

Webmail:
/var/db/squirrelmail/data/.

Mailman List files:
/var/spool/mailman
/usr/share/mailman

/var/named (DNS zone files)

SEDS Custom Startup Scripts:
/Library/StartupItems/
/Library/LaunchDaemons/

/backups/ (Archive of back sets of Mysql and other things)


There are many files and some directories in /etc that are relevant:
/etc/postfix/ (Some of this will change with new mail config)
/etc/watchdog.conf (Processes to spawn and monitor)
/etc/certificates/ (Home brew SSL certificates)
/etc/imapd.conf
/etc/named.conf
/etc/sudoers


And lots of other things.  In general, this whole directory should be copied to a 
migration directory and picked through for relevant stuff to migrate.

Stuff to put in an exclude file for rsync: 

.Spotlight-V100/
.Trash/
core
Cache
cache

Other stuff *NOT* to copy (will be re-built, re-installed) 

/Applications/
/Developer/
/Library/
/System/
/usr/
/sbin/
/sw/
/man/
/dev/
/bin/
/cores/
/Volumes/
/Network/

Configure Services on New Server

User Accounts

Chris followed these fine instructions verbatim to successfully copy over the accounts and passwords from the old server. Files are in /migration/opendirectory/

MySQL Databases

Aaron Schultz to provide details.

Take old Server Offline

Disable Web Server 

/opt/local/apache2/bin/apachectl stop

Disable Email Services 

serveradmin mail stop

Stop FTP Server 

serveradmin ftp stop

Stop File Sharing 

serveradmin afp stop

Prevent User Logins 

Edit the file: /etc/nologin  with the message to send users  (see man login)

Perform Final Synchronization

MySQL Databases

Aaron Schultz to provide details.

Update DNS Records to make new server Prime

As the day gets closer, lower the TTL of the DNS records so the switchover can be fast. It's already only 15m for the TTL and 2H for expire (for the SEDS.org domain), but we can cut this down even more as it gets real close.

Will also need to update the primary DNS records for:

yurisnight.net (Chris) nasa-academy.org (Chris) uk2.seds.org (Chris) lewicki.com (Chris) protoforge.org (Aaron) protoforge.net (Aaron) spacevision.org (Josh) mountainviewgardens.com (Guy)

The slaves can all be left alone (configure is elsewhere). Once we're all done with this, we'll do some DNS sharing between the development and prime servers.

It would be ideal to get all the DNS configuration into the Server Admin GUI for easier maintenance.

Rebuild old Server as Development Platform

Yeah, let's do that.

Misc Things to Fix

  • ~/bin/process-weblog.sh calls gnu date (used to be in GNU fileutils in /usr/local/bin, installed mac ports coreutils, but that doesn't have it).
    • Called by cron daily.
  • A few mailman Icons weren't present in the macports apache2 install
    • cp -ivn /usr/share/httpd/icons/* /opt/local/apache2/icons/ 
/usr/share/httpd/icons/PythonPowered.png -> /opt/local/apache2/icons/PythonPowered.png
/usr/share/httpd/icons/gnu-head-tiny.jpg -> /opt/local/apache2/icons/gnu-head-tiny.jpg
/usr/share/httpd/icons/mailman-large.jpg -> /opt/local/apache2/icons/mailman-large.jpg
/usr/share/httpd/icons/mailman.jpg -> /opt/local/apache2/icons/mailman.jpg
/usr/share/httpd/icons/mm-icon.png -> /opt/local/apache2/icons/mm-icon.png
  • Edit PHP path in /web/seds/support/html/cli/index.php to point to /opt/local/bin/php
  • Create a robots.txt file for every configuration virtual site (produces lots of lines in the error_log files). This will just update the timestamp on the ones that are already there: 
grep "<Directory " /opt/local/apache2/conf/sites/* | grep "/web"  | perl -pe "s/^.*Directory \"//" | perl -pe "s/\">$/\/robots.txt/" | xargs touch
  • Update /web/seds/wiki/html/LocalSettings.php to change from single quotes to double quotes in Bad Behavior load 
include_once( "$IP/includes/DatabaseFunctions.php" );
include( "$IP/extensions/Bad-Behavior/bad-behavior-mediawiki.php" );
  • Also upgraded SpamBlacklist and BadBehavior extensions to get rid of some PHP compatibility errors in the older code.
  • Need to get the htimage maps in /web/seds/org/html/Maps/*.html converted from htimage to the more recent MAP declaration (perhaps ask Spider/Hartmut Frommert)
  • Need to get the http://www.seds.org/archive/nodes/smallindex.html and other node archive pages pointing to purple/red/blue/yellow/green ball.gif pictures in an accessible location. Perhaps make an /icons/archive directory for these.
Retrieved from "http://wiki.seds.org/index.php/Server_Upgrade"
Personal tools