How To Migrate Linux Servers Part 2 - Transfer Core Data

How To Migrate Linux Servers Part 2 - Transfer Core Data

Introduction

There are many scenarios where you might have to move your data and operating requirements from one server to another. You may need to implement your solutions in a new datacenter, upgrade to a larger machine, or transition to new hardware or a new VPS provider.

Whatever your reasons, there are many different considerations you should make when migrating from one system to another. Getting functionally equivalent configurations can be difficult if you are not operating with a configuration management solution such as Chef, Puppet, or Ansible. You need to not only transfer data, but also configure your services to operate in the same way on a new machine.

In the last article, we prepped our servers for data migration. At this point, your target and source system should be able to communicate (the target system should have SSH access to the source system). You should also have a list of software and services that you need to transfer, complete with version numbers of the most important components.

In this guide, we'll continue where we left off and begin the actual migration to our new server.

General Strategy

Before we begin, we should outline our general strategy for migrating data from our source to our target system.

The general idea is to transfer all of the relevant pieces of information while leaving the target system as clean as possible.

Some migration strategies simply point rsync at the root of the source machine and then pass in some exclude lines to tell the process not to include files that we know will cause conflicts. We won't take this approach. Migrating large pieces of system data onto a live operating system can cause unpredictable results and we want to end up with a stable system.

Not only that, but we don't want to needlessly clutter our new system with files that are no longer relevant to our operational requirements. This will take more effort, but it will lead to a more usable and friendly configuration when we are finished.

So we won't be just migrating every possible non-conflicting file to the new system if it isn't relevant to what we are hoping to achieve. Instead, we will be deciding exactly which data needs to be moved as a functional requirement for our purposes. This includes data and configuration details, user, jobs, etc.

Creating a Migration Script

We will be making these decisions as we go, and adding them to a migration script.

This will give you a number of important advantages. It will allow you to easily re-run the commands again if there is a problem or in order to capture data changes on the source system after the first run. It will self-document the commands you used to transfer the data. It will also allow your source server to continue onto the next item of data transfer without user interaction.

As you write the script, you should be able to run it multiple times, refining it as you go. Most of the files will by transferred through rsync, which will only transfer file changes. If the other data transfer portions take a long time, you can safely comment them out until you are fairly sure your script is in its final state.

This article will mostly be a guide on what to add to your migration script to make your migration successful. It will provide general guidelines more often than specifics.

We can create a simple migration script in the root user's home directory on the target system. We will use this to automate a large portion of our data migration operations:

nano /root/sync.sh

Inside the file, begin with a standard script heading (we will use "sh" to make this more portable, but you can use "bash" if you would like to use the extended features it offers and have it available on both systems):

#!/bin/sh

We will add to this as we continue. For now though, let's exit the file quickly so that we can make it executable.

Back on the command line, make the script executable by typing:

chmod 700 /root/sync.sh

To run the script at any time, you can now call it using its absolute path:

/root/sync.sh

Or its relative path:

cd /root
./sync.sh

You should test the script regularly as you go along to see if there are issues that come up.

Install Needed Programs and Services

The first step that we need to take prior to automation is to acquire the packages that you need to get these services up and running. We could also add this to the script, but it is easier to just do this portion by hand and document it in our script.

The configuration details will come later. For now, we need these applications installed and basic access configured so that we can get to work. You should have a list of required packages and versions from your source machine.

Add Additional Repositories if Necessary

Before we attempt to get these versions from our package manager, we should inspect our source system to see if any additional repositories have been added.

On Ubuntu/Debian machines, you can see if alternative software sources are present on your source system by investigating a few locations:

nano /etc/apt/sources.list

This is the main source list. Additional source lists can be contained in the sources.list.d directory:

ls /etc/apt/sources.list.d

If you need to, add the same sources to your target machine to have the same package versions available.

On a RHEL-based system, you can use yum to list the repositories configured for the server:

yum repolist enabled

You can then add additional repositories to your target system by typing:

yum-config-manager --add-repo repo_url

If you make any changes to your source list, add them as comments at the top of your migration script. This way, if you have to start from a fresh install, you will know what procedures need to happen before attempting a new migration.

nano /root/sync.sh

#!/bin/sh

#############
# Prep Steps
#############

# Add additional repositories to /etc/apt/source.list
#       deb http://example.repo.com/linux/deb stable main non-free

Save and close the file.

Specifying Version Constraints and Installing

You now have the repositories updated to match your source machine.

On Ubuntu/Debian machines, you can now attempt to install the version of the software that you need on your target machine by typing:

apt-get update
apt-get install package_name=version_number

Many times, if the version of the package is older, it will have been removed from the official repositories. In this case, you may have to manually hunt down the older version of the .deb files and their dependencies and install them manually with:

dpkg -i package.deb

This is necessary if matching the software version is important for your application. Otherwise, you can just install regularly with your package manager.

For RHEL-based systems, you can install specific versions of software by typing:

yum install package_name-version_number

If you need to hunt down rpm files that have been removed from the repository in favor of newer versions, you can install them with yum after you've found them like this:

yum install package_name.rpm

Install any relevant software that is available from your package manager into the new system. In the event that the software you need is not available through a repository or other easy means and has been installed by source or pulled in as a binary from a project's website, you will have to replicate this process on the target system.

Again, keep track of what operations you are performing here. We will include them as comments in a script we are creating:

nano /root/sync.sh

#!/bin/sh

#############
# Prep Steps
#############

# Add additional repositories to /etc/apt/source.list
#       deb http://example.repo.com/linux/deb stable main non-free

# Install necessary software and versions
#       apt-get update
#       apt-get install apache2=2.2.22-1ubuntu1.4 mysql-server=5.5.35-0ubuntu0.12.04.2 libapache2-mod-auth-mysql=4.3.9-13ubuntu3 php5-mysql=5.3.10-1ubuntu3.9 php5=5.3.10-1ubuntu3.9 libapache2-mod-php5=5.3.10-1ubuntu3.9 php5-mcrypt=5.3.5-0ubuntu1

Again, save and close the file.

Start Transferring Data

The actual transfer of data can easily be the most time-intensive part of the migration. If you are migrating a server with a lot of data, it is probably a good idea to start transferring data sooner rather than later. You can refine your commands later on, and rsync only transfers the differences between files, so this shouldn't be a problem.

We can begin by starting an rsync of any large chunks of user data that need to be transferred. In this context, we are using "user" data to refer to any significant data needed by your server except database data. This includes site data, user home directories, configuration files, etc.

Installing and Using Screen

To do this effectively, we're going to want to start a screen session on our target system that you can leave running while you continue to work.

You can install screen using your distribution's package manager. On Ubuntu or Debian, you could type this:

apt-get update
apt-get install screen

You can find out how to operate screen by checking out this link.

Basically, you need to start a new screen session like this on your target server:

screen

A screen session will start, and drop you back into a command line. It will probably look like nothing has happened, but you're now operating a terminal that is contained within the screen program.

All of the work that we will do during our migration will happen within a screen session. This allows us to easily jump between multiple terminal sessions, and allows us to pick up where we left off if we have to leave our local terminal or we get disconnected.

You can issue commands here and then disconnect the terminal, allowing it to continue running. You can disconnect at any time by typing:

CTRL-a d

You can reconnect later by typing:

screen -r

If you need to create another terminal window within your screen session, type:

CTRL-a c

To switch between windows, type these two to cycle through windows in either direction:

CTRL-a n
CTRL-a p

Destroy a window by typing:

CTRL-a k

Begin File Transfers Early

Inside of your screen session, start any rsync tasks that you anticipate taking a long time to complete. The time scale here depends on the amount of significant (non-database) data you have to transfer.

The general command you'll want to use is:

rsync -avz --progress source_server:/path/to/directory/to/transfer /path/to/local/directory

You can find out more about how to create appropriate rsync commands by reading this article. You may have to create the directories leading up to the destination in order for the command to execute properly.

When you have your rsync session running, create a new screen window and switch to it by typing:

CTRL-a c

Check back periodically to see if the syncing is complete and perhaps to start a subsequent sync by typing:

CTRL-a p

Adjusting the Script to Sync Data and Files

Now, you should add the same rsync command that you just executed into the script you are creating. Add any additional rsync commands that you need in order to get all of your significant user and application data onto your target server.

We will not worry about database files at this point, because there are better methods of transferring those files. We will discuss these in a later section.

#!/bin/sh

#############
# Prep Steps
#############

# Add additional repositories to /etc/apt/source.list
#       deb http://example.repo.com/linux/deb stable main non-free

# Install necessary software and versions
#       apt-get update
#       apt-get install apache2=2.2.22-1ubuntu1.4 mysql-server=5.5.35-0ubuntu0.12.04.2 libapache2-mod-auth-mysql=4.3.9-13ubuntu3 php5-mysql=5.3.10-1ubuntu3.9 php5=5.3.10-1ubuntu3.9 libapache2-mod-php5=5.3.10-1ubuntu3.9 php5-mcrypt=5.3.5-0ubuntu1

#############
# File Transfer
#############


# Rsync web root
rsync -avz --progress 111.222.333.444:/var/www/site1 /var/www/

# Rsync the apache configuration files
rsync -avz --progress 111.222.333.444:/etc/apache2/* /etc/apache2/

# Rsync php configuration
rsync -avz --progress 111.222.333.444:/etc/php5/* /etc/php5/

# Rsync mysql config files
rsync -avz --progress 111.222.333.444:/etc/mysql/* /etc/mysql/

# Rsync home directories
. . .

You should add any rsync commands that you need to transfer your data and configurations off of the source system.

This does not need to be perfect, because we can always go back and adjust it, so just try your best. If you're unsure of whether you need something right now, leave it out for the time being and just add a comment instead.

We will be running the script multiple times, allowing you to modify it to pick up additional files if you end up needing them. Being conservative about what you transfer will keep your target system clean of unnecessary files.

We are trying to replicate the functionality and data of the original system, and not necessarily the mess.

Modifying Configuration Files

Although many pieces of software will work exactly the same after transferring the relevant configuration details and data from the original server, some configuration will likely need to be modified.

This presents a slight problem with our syncing script. If we run the script to sync our data, and then modify the values to reflect the correct information for its new home, these changes will be wiped out the next time we run the script again.

Remember, we will likely be running the rsync script multiple times to catch up with changes that have occurred on the source system since we've started our migration. The source system can change significantly during the course of migrating and testing the new server.

There are two general paths that we can take to avoid wiping out our changes. First, I'll discuss the easy way, and follow up with what I consider the more robust solution.

The Quick and Dirty Way

The easy way of addressing this is to modify the files as needed on the target system after the first sync operation. Afterwards, you then can modify the rsync commands in your script to exclude the files that you adjusted.

This will cause rsync to not sync these files on subsequent runs, which would overwrite your changes with the original files again.

This can be accomplished by commenting out the previous sync command and adding a new one with some exclude statements like this:

# rsync -avz --progress 111.222.333.444:/etc/mysql/* /etc/mysql/
rsync -avz --progress --exclude='my.cnf' 111.222.333.444:/etc/mysql/* /etc/mysql/

You should add exclusion lines for any files under the rsync directory specification that have been modified. It would also be a good idea to add a comment as to what was modified in the file, in case you actually do need to recreate it at any point.

# Adding exclude rule.  Changed socket to '/mysqld/mysqld.sock'
# rsync -avz --progress 111.222.333.444:/etc/mysql/* /etc/mysql/
rsync -avz --progress --exclude='my.cnf' 111.222.333.444:/etc/mysql/* /etc/mysql/

The Robust and Recommended Way

While the above method addresses the problem in some ways, it's really just avoiding the issue instead of solving it. We can do better.

Linux systems include a variety of text manipulators that are very useful for scripting. In fact, most of these programs are made specifically to allow their use in a scripted environment.

The two most useful utilities for this task are sed and awk. 

The basic idea is that we can script any changes that we would be making manually, so that the script itself will perform any necessary modifications.

So in the previous example, instead of adding an exclusion for the file we modified after the fact, we could keep that rsync command and make that change automatically using a sed command:

rsync -avz --progress 111.222.333.444:/etc/mysql/* /etc/mysql/

# Change socket to '/mysqld/mysqld.sock'
sed -i 's_/var/run/mysqld/mysqld.sock_/mysqld/mysqld.sock_g' /etc/mysql/my.cnf

This will change the socket location in every instance of the file, each time the file is transferred. Make sure that the text manipulation lines come after the lines that sync the files that they operate on.

In a similar way, we can easily script changes made to tabular data files using awk. For instance, the /etc/shadow file is divided into tabs delimited by the colon (:) character. We could use awk to remove the hashed root password from the second column like this:

awk 'BEGIN { OFS=FS=":"; } $1=="root" { $2=""; } { print; }' /etc/shadow > shadow.tmp && mv shadow.tmp /etc/shadow && rm shadow.tmp

This command is telling awk that both the original and the output delimiter should be ":" instead of the default space. We then specify that if column 1 is equal to "root", then column 2 should be set to an empty string.

Up until fairly new versions of awk, there was no option to edit in place, so here we are writing this file to a temporary file, overwriting the original file, and then removing the temporary file.

We should do our best to script all of the changes needed in our files. This way, it will be easy to reuse some of the lines from our migration script for other migrations, with some easy modification.

An easy way of doing this is to go through your script and add comments to your script for each file that needs to be modified. After you know your requirements, go back and add the commands that will perform the necessary operations.

Add these changes to your script and let's move on.

Dump and Transfer your Database files

If your system is using a database management system, you will want to dump the database using the methods available for your system. This will vary depending on the DBMS you use (MySQL, MariaDB, PostgreSQL, etc.).

For a regular MySQL system, you can export the database using something like this:

mysqldump -Q -q -e -R --add-drop-table -A -u root -proot_password > /root/database_name.db

MySQL dump options are highly dependent on the context, so you'll have to explore which options are right for your system before deciding. This is beyond the scope of this article.

Let's go over what these options will do for the database dump.

• -Q: This option is enabled by default, but is added here for extra safety. It puts identifiers like database names inside quotes to avoid misinterpretation.
• -q: This stands for quick and can help speed up large table dumps. In actuality, it is telling MySQL to operate on a row-by-row basis instead of trying to handle the entire table at once.
• -e: This creates smaller dump files by grouping insert statements together instead of handling them individually when the dump file is loaded.
• -R: This allows MySQL to also dump stored routines along with the rest of the data.
• --add-drop-table: This option specifies that MySQL should issue a DROP TABLE command prior to each CREATE TABLE to avoid running into an error if the table already exists.
• -A: This option specifies that MySQL should dump all of the databases.
• -u: This details the MySQL user to use for the connection. This should be root.
• -p: This is the password needed for the MySQL root account.

This will create a MySQL dump of the source system's MySQL data on the original system. We can wrap this in an SSH command to have it execute remotely:

ssh root@111.222.333.444 'mysqldump -Q -q -e -R --add-drop-table -A -u root -proot_password > /root/database_name.db'

We can then use a normal rsync command to retrieve the file when it is finished:

rsync -avz --progress 111.222.333.444:/root/database_name.db /root/

After that, we can import the dump into the target system's MySQL instance:

mysql -u root -proot_password < /root/database_name.db

Another option is to configure a replication setup between the original database and the target system's database. This can allow you to simply swap the master and the slave when you are finished, in order to finalize the database migration.

If you go this route, make sure to add comments to your script specifying your configuration. If there is a big issue, you want to be able to have good information on what you did so that you can avoid it on a second attempt.

Next Steps

You should now have most of your data on your target system, or be in the process of transferring. This will accomplish the bulk of actual data transfer, but we still need to do quite a bit of configuration on our system to match our previous machine.

In the next article, we will move on to transfer other information and user settings.

How To Migrate Linux Servers Part 1 - System Preparation

How To Migrate Linux Servers Part 1 - System Preparation

Introduction

There are many scenarios where you might have to move your data and operating requirements from one server to another. You may need to implement your solutions in a new datacenter, upgrade to a larger machine, or transition to new hardware or a new VPS provider.

Whatever your reasons, there are many different considerations you should make when migrating from one system to another. Getting functionally equivalent configurations can be difficult if you are not operating with a configuration management solution such as Chef, Puppet, or Ansible. You need to not only transfer data, but also configure your services to operate in the same way on a new machine.

In this guide, we will discuss how to prepare your source and target systems for a migration. This will include getting your two machines to communicate with SSH keys and a heavy investigation as to what components need to be transferred. We will work on the actual migration in the next article.

Make Backups

The first step to take when performing any potentially destructive step is to create fresh backups. Just because there shouldn't be a problem doesn't mean that something unexpected isn't going to happen. You don't want to be left in a situation where a command breaks something on your current production machine before the replacement is up and running.

There are a number of different ways to back up your server. Your selection will depend on what options make sense for your scenario and what you are most comfortable with.

If you have access to the physical hardware and a space to backup (disk drive, USB, etc), you can clone the disk using any one of the many image backup solutions available. A functional equivalent when dealing with VPS machines is to take a snapshot or image from within the control panel interface.

Other options often will preserve data and perhaps some of the information about your services. It all depends on which backup mechanism you wish to implement. Our community pages have articles on many different backup options. Choose one that makes sense for your data and system.

Once you have completed backups, you are ready to continue. For the remainder of this guide, we will assume that you are logged into both systems as root.

Gather Information about the Source System

Before we begin migrating, we should take the initial steps to set up our target system to match our source system.

We will want to match as much as we can between the current server and the one we plan on migrating to. If you want the migration to go smoothly, then you shouldn't take this as an opportunity to upgrade to the newest version or try new things. Making changes can lead to instability and problems down the line.

Most of the basic information that will help you decide which server system to create for the new machine can be retrieved with a simple uname command:

uname -r

3.2.0-24-virtual

This is the version of the kernel that our current system is running. In order to make things go smoothly, it's always a good idea to try to match that on the target system.

uname -m

i686

This is the system architecture. i686 indicates that this is a 32-bit system. If the returned string was x86_64, this would mean that this is a 64-bit system.

You should also try to match the distribution and version of your source server. If you don't know the version of the distribution that you have installed on the source machine, you can find out by typing:

cat /etc/issue

Ubuntu 12.04.2 LTS \n \l

You should create your new server with these same parameters if possible. In this case, we would create a 32-bit Ubuntu 12.04 system. If possible, we'd also attempt to match the kernel version on the new system.

Set Up SSH Key Access between Source and Target Servers

We'll need our servers to be able to communicate so that they can transfer files. The easiest way to do this is with SSH keys. 

We want to create a new key on our target server so that we can add that to our source server's authorized_keys file. This is cleaner than the other way around, because then the new server will not have a stray key in its authorized_keys file when the migration is complete.

First, on your destination machine, check that your root user doesn't already have an SSH key (you should be logged in as root) by typing:

ls ~/.ssh

authorized_keys

If you see files called id_rsa.pub and id_rsa, then you already have keys and you'll just need to transfer them.

If you don't see those files, create a new key pair by typing:

ssh-keygen -t rsa

Press "Enter" through all of the prompts to accept the defaults.

Now, transfer the key to the source server by typing:

cat ~/.ssh/id_rsa.pub | ssh other_server_ip "cat >> ~/.ssh/authorized_keys"

You should now be able to SSH freely to your source server from the target system by typing:

ssh other_server_ip

You should not be prompted for a password if you configured this correctly.

Create a List of Requirements

This is actually the first part where you're going to be doing in-depth analysis of your system and requirements.

During the course of operations, your software requirements can change. Sometimes old servers have some services and software that were needed at one point, but have been replaced.

While unneeded services should be disabled and, if completely unnecessary, uninstalled, this doesn't always happen. You need to discover what services are being used on your source server, and then decide if those services should exist on your new server.

The way that you discover services and runlevels largely depends on the type of "init" system that your server employs. The init system is responsible for starting and stopping services, either at the user's command or automatically.

Discovering Services and Runlevels on System V Servers

System V is one of the older init systems still in use on many servers today. Ubuntu has attempted to switch to the Upstart init system, and in the future may be transitioning to a Systemd init system.

Currently, both System V style init files and the newer Upstart init files can be found on the same systems, meaning you'll have more places to look. Other systems use System V as well. You can see if your server uses System V by typing:

which service

/usr/sbin/service

If the command returns a system path, as it did above, you have System V on your system.

You can get an idea of which services are currently running by typing this:

service --status-all

 [ ? ]  acpid
 [ ? ]  anacron
 [ + ]  apache2
 [ ? ]  atd
 [ - ]  bootlogd
 [ ? ]  console-setup
 [ ? ]  cron
 [ ? ]  cryptdisks
 . . .

This will list all of the services that the System-V init system knows about. The "+" means that the service is started, the "-" means it is stopped, and the "?" means that System-V doesn't know the state of the service.

If System-V doesn't know the state of the service, it's possible that it is controlled by an alternative init system. On Ubuntu systems, this is usually Upstart.

Other than figuring out which services are running currently, another good piece of information to have is what runlevel a service is active in. Runlevels dictate which services should be made available when the server is in different states. You will probably want to match the source server's configuration on the new system.

You can discover the runlevels that each service will be active for using a number of tools. One way is through tools like chkconfig or sysv-rc-conf.

On an Ubuntu or Debian system, you can install and use chkconfig to check for which System V services are available at different runlevels like this. Most RHEL-based systems should already have this software installed:

apt-get update
apt-get install chkconfig
chkconfig --list

acpid                     0:off  1:off  2:off  3:off  4:off  5:off  6:off
anacron                   0:off  1:off  2:off  3:off  4:off  5:off  6:off
apache2                   0:off  1:off  2:on   3:on   4:on   5:on   6:off
atd                       0:off  1:off  2:off  3:off  4:off  5:off  6:off
bootlogd                  0:off  1:off  2:off  3:off  4:off  5:off  6:off
console-setup             0:off  1:off  2:off  3:off  4:off  5:off  6:off
cron                      0:off  1:off  2:off  3:off  4:off  5:off  6:off
cryptdisks                0:on   1:off  2:off  3:off  4:off  5:off  6:off
cryptdisks-early          0:on   1:off  2:off  3:off  4:off  5:off  6:off
. . .

Another alternative is sysv-rc-conf, which can be installed and run like this:

apt-get update
apt-get install sysv-rc-conf
sysv-rc-conf --list

acpid       
anacron     
apache2      0:off  1:off   2:on    3:on    4:on    5:on    6:off
atd         
bootlogd    
console-setu
cron        
cryptdisks   0:on   6:on
cryptdisks-e 0:on   6:on
. . .

If you would like to manually check instead of using a tool, you can do that by checking a number of directories that take the form of /etc/rc*.d/. The asterisk will be replaced with the number of the runlevel.

For instance, to see what services are activated by System V in runlevel 2, you can check the files there:

cd /etc/rc2.d
ls -l

total 4
-rw-r--r-- 1 root root 677 Jul 26  2012 README
lrwxrwxrwx 1 root root  18 Dec 28  2012 S20php5-fpm -> ../init.d/php5-fpm
lrwxrwxrwx 1 root root  15 Apr 26  2012 S50rsync -> ../init.d/rsync
lrwxrwxrwx 1 root root  14 Jun 21  2013 S75sudo -> ../init.d/sudo
lrwxrwxrwx 1 root root  17 Dec 28  2012 S91apache2 -> ../init.d/apache2
. . .

These are links to configuration files located in /etc/init.d/. Each link that begins with an "S" means that it is used to start a service. Scripts that start with a "K" kill services off at that runlevel.

Discovering Services and Runlevels on Upstart Servers

Ubuntu and Ubuntu-based servers are pretty much the only servers that implement the Upstart init system by default. These are typically used as the main init system, with System V being configured for legacy services.

To see if your server has an Upstart init system, type:

which initctl

/sbin/initctl

If you receive a path to the executable as we did above, then your server has Upstart capabilities and you should investigate which services are controlled by Upstart.

You can see which services are started by Upstart by typing:

initctl list

mountall-net stop/waiting
passwd stop/waiting
rc stop/waiting
rsyslog start/running, process 482
tty4 start/running, process 728
udev start/running, process 354
upstart-udev-bridge start/running, process 350
ureadahead-other stop/waiting
. . .

This will tell you the current state of all Upstart managed services. You can tell which services are being run currently and maybe see if there are services that provide the same functionality where one has taken over for a legacy service that is no longer in use.

Again, you should become familiar with what services are supposed to be available at each runlevel.

You can do this with the initctl command by typing:

initctl show-config

mountall-net
  start on net-device-up
passwd
  start on filesystem
rc
  emits deconfiguring-networking
  emits unmounted-remote-filesystems
  start on runlevel [0123456]
  stop on runlevel [!$RUNLEVEL]
rsyslog
  start on filesystem
  stop on runlevel [06]
  . . .

This spits out a lot of configuration information for each service. The part to look for is the runlevel specification.

If you would rather gather this information manually, you can look at the files located in the /etc/initdirectory (notice the omission of the ".d" after the "init" here).

Inside, you will find a number of configuration files. Within these files, there are runlevel specifications given like this:

start on runlevel [2345]
stop on runlevel [!2345]

You should have a good idea of different ways of discovering Upstart services and runlevels.

Discovering Services and Runlevels on Systemd Servers

A newer init style that is increasingly being adopted by distributions is the systemd init system.

Systemd is rather divergent from the other types of init systems, but is incredibly powerful. You can find out about running services by typing:

systemctl list-units -t service

UNIT                                           LOAD   ACTIVE SUB     DESCRIPTION
atd.service                                    loaded active running ATD daemon
avahi-daemon.service                           loaded active running Avahi mDNS/DNS-SD Stack
colord.service                                 loaded active running Manage, Install and Generate Color Profiles
cups.service                                   loaded active running CUPS Printing Service
dbus.service                                   loaded active running D-Bus System Message Bus
dcron.service                                  loaded active running Periodic Command Scheduler
dkms.service                                   loaded active exited  Dynamic Kernel Modules System
. . .

Systemd doesn't exactly replicate the runlevels concept of other init systems. Instead, it implements the concept of "targets". While systems with traditional init systems can only be in one runlevel at a time, a server that uses systemd can reach several targets at the same time.

Because of this, figuring out what services are active when is a little bit more difficult.

You can see which targets are currently active by typing:

systemctl list-units -t target

UNIT                LOAD   ACTIVE SUB    DESCRIPTION
basic.target        loaded active active Basic System
cryptsetup.target   loaded active active Encrypted Volumes
getty.target        loaded active active Login Prompts
graphical.target    loaded active active Graphical Interface
local-fs-pre.target loaded active active Local File Systems (Pre)
local-fs.target     loaded active active Local File Systems
. . .

You can list all available targets by typing:

systemctl list-unit-files -t target

UNIT FILE                 STATE   
basic.target              static  
bluetooth.target          static  
cryptsetup.target         static  
ctrl-alt-del.target       disabled
default.target            disabled
emergency.target          static  
final.target              static  
getty.target              static  
graphical.target          disabled
halt.target               disabled
. . .

From here, we can find out which services are associated with each target. Targets can have services or other targets as dependencies, so we can see what policies each target implements by typing:

systemctl list-dependencies target_name.target

For instance, you might type something like this:

systemctl list-dependencies multi-user.target

multi-user.target
├─atd.service
├─avahi-daemon.service
├─cups.path
├─dbus.service
├─dcron.service
├─dkms.service
├─gpm.service
. . .

This will list the dependency tree of that target, giving you a list of services and other targets that get started when that target is reached.

Double Checking Services Through Other Methods

While most services will be configured through the init system, there are possibly some areas where a process or service will slip through the cracks and be controlled independently.

We can try to find these other services and processes by looking at the side effects of these services. In most cases, services communicate with each other or outside entities in some way. There are only a specific number of ways that services can communicate, and checking those interfaces is a good way to spot other services.

One tool that we can use to discover network ports and Unix sockets that are being used by processes to communicate is netstat. We can issue a command like this to get an overview of some of our services:

netstat -nlp

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      762/mysqld      
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      832/apache2     
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      918/sshd        
tcp        0      0 127.0.0.1:9000          0.0.0.0:*               LISTEN      799/php-fpm.conf)
tcp6       0      0 :::22                   :::*                    LISTEN      918/sshd        
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node   PID/Program name    Path
unix  2      [ ACC ]     STREAM     LISTENING     1526     1/init              @/com/ubuntu/upstart
unix  2      [ ACC ]     SEQPACKET  LISTENING     1598     354/udevd           /run/udev/control
unix  2      [ ACC ]     STREAM     LISTENING     6982     480/dbus-daemon     /var/run/dbus/system_bus_socket
unix  2      [ ACC ]     STREAM     LISTENING     8378     762/mysqld          /var/run/mysqld/mysqld.sock
unix  2      [ ACC ]     STREAM     LISTENING     1987     746/acpid           /var/run/acpid.socket

The port numbers in the first section are associated with the programs on the far right. Similarly, the bottom portion focuses on Unix sockets that are being used by programs.

If you see services here that you do not have information about through the init system, you'll have to figure out why that is and what kind of information you'll need to gather about that service.

You can get similar information about the ports services are making available by using the lsofcommand:

lsof -nPi

COMMAND   PID     USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
mysqld    762    mysql   10u  IPv4   8377      0t0  TCP 127.0.0.1:3306 (LISTEN)
php5-fpm  799     root    6u  IPv4   8195      0t0  TCP 127.0.0.1:9000 (LISTEN)
php5-fpm  800 www-data    0u  IPv4   8195      0t0  TCP 127.0.0.1:9000 (LISTEN)
php5-fpm  801 www-data    0u  IPv4   8195      0t0  TCP 127.0.0.1:9000 (LISTEN)
php5-fpm  802 www-data    0u  IPv4   8195      0t0  TCP 127.0.0.1:9000 (LISTEN)
php5-fpm  803 www-data    0u  IPv4   8195      0t0  TCP 127.0.0.1:9000 (LISTEN)
apache2   832     root    3u  IPv4   8210      0t0  TCP *:80 (LISTEN)
sshd      918     root    3r  IPv4   7738      0t0  TCP *:22 (LISTEN)
. . .

You can get some great information from the ss command on what processes are using what ports and sockets:

ss -nlpaxtudw

Netid  State      Recv-Q Send-Q   Local Address:Port     Peer Address:Port 
u_str  LISTEN     0      0      @/com/ubuntu/upstart 1526                * 0     users:(("init",1,7))
u_str  ESTAB      0      0      @/com/ubuntu/upstart 1589                * 0     users:(("init",1,10))
u_str  ESTAB      0      0                    * 1694                * 0     users:(("dbus-daemon",480,6))
u_str  ESTAB      0      0                    * 1695                * 0     users:(("dbus-daemon",480,7))
u_str  ESTAB      0      0                    * 1803                * 0

Gathering Package Versions

After all of that exploration, you should have a good idea about what services are running on your source machine that you should be implementing on your target server.

You should have a list of services that you know you will need to implement. For the transition to go smoothly, it is important to attempt to match versions where ever it is possible.

You obviously won't be able to go through every single package installed on the source system and attempt to replicate it on the new system, but you should check the software components that are important for your needs and try to find the version number.

You can try to get version numbers from the software itself, sometimes by passing -v or --versionflags to the commands, but usually this is easier to accomplish through your package manager. If you are on an Ubuntu/Debian based system, you can see which version of the packages are installed from the package manager by typing:

dpkg -l | grep package_name

If you are instead on a RHEL-based system, you can use this command to check the installed version instead.

rpm -qa | grep package_name

This will give you a good idea of the program version you are looking to installed.

Keep a list of the version numbers of the important components that you wish to install. We will attempt to acquire these on the target system.

Next Steps

You should now have a good idea of what processes and services on your source server need to be transferred over to your new machine. You should also have the preliminary steps completed to allow your two server instances to communicate with each other.

The groundwork for your migration is now complete. You should now be able to jump into the migration in the next article with a good idea of what you need to do.