Tuesday, February 14, 2017

Setup Linux Container with LXC on CentOS 7 / RHEL 7

Setup Linux Container with LXC on CentOS 7 / RHEL 7

LXC is a free software, it is released under GNU LGPLv2.1+ license. LXC project is sponsored by Canonical Ltd who is behind Ubuntu OS.
In this guide, I will show you how to install LXC, and how to create and manage LXC using the command line.

Prerequisites:

LXC is not available on Base repo. so we need to add EPEL repository to the server using the following command.
# yum -y install epel-release
LXC containers use bridge networking to have access to / from an external network, before starting the container we must create a network bridge on CentOS 7 / RHEL 7. The name of the network bridge should be “virbr0”

Install LXC on CentOS 7:

Once you have completed prerequisites, it’s a time to install LXC. Install the LXC and other important packages required for the containers to work properly.
# yum -y install lxc lxc-templates libcap-devel libcgroup busybox wget bridge-utils
Run the following command to check everything is fine to run containers.
# lxc-checkconfig

 Kernel configuration not found at /proc/config.gz; searching...
 Kernel configuration found at /boot/config-3.10.0-327.el7.x86_64
 --- Namespaces ---
 Namespaces: enabled
 Utsname namespace: enabled
 Ipc namespace: enabled
 Pid namespace: enabled
 User namespace: enabled
 Network namespace: enabled
 Multiple /dev/pts instances: enabled
--- Control groups ---
 Cgroup: enabled
 Cgroup clone_children flag: enabled
 Cgroup device: enabled
 Cgroup sched: enabled
 Cgroup cpu account: enabled
 Cgroup memory controller: enabled
 Cgroup cpuset: enabled
--- Misc ---
 Veth pair device: enabled
 Macvlan: enabled
 Vlan: enabled
 Bridges: enabled
 Advanced netfilter: enabled
 CONFIG_NF_NAT_IPV4: enabled
 CONFIG_NF_NAT_IPV6: enabled
 CONFIG_IP_NF_TARGET_MASQUERADE: enabled
 CONFIG_IP6_NF_TARGET_MASQUERADE: enabled
 CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled
--- Checkpoint/Restore ---
 checkpoint restore: enabled
 CONFIG_FHANDLE: enabled
 CONFIG_EVENTFD: enabled
 CONFIG_EPOLL: enabled
 CONFIG_UNIX_DIAG: enabled
 CONFIG_INET_DIAG: enabled
 CONFIG_PACKET_DIAG: enabled
 CONFIG_NETLINK_DIAG: enabled
 File capabilities: enabled
Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig

Creating Linux Containers:

LXC comes with ready-made templates for easy installation of containers, you can list down the available templates using the following command.
# ls /usr/share/lxc/templates/

 lxc-alpine    lxc-archlinux  lxc-centos  lxc-debian    lxc-fedora  lxc-openmandriva  lxc-oracle  lxc-sshd    lxc-ubuntu-cloud
 lxc-altlinux  lxc-busybox    lxc-cirros  lxc-download  lxc-gentoo  lxc-opensuse      lxc-plamo   lxc-ubuntu
To create a container, issue the following command.
# lxc-create -n centos_lxc -t centos
Where,
-n <container name>
-t <template>
Once you have issued the above command, LXC will start creating the container with name “centos_lxc”
Sample output:
Host CPE ID from /etc/os-release: cpe:/o:centos:centos:7
 Checking cache download in /var/cache/lxc/centos/x86_64/7/rootfs ...
 Downloading centos minimal ...
 Loaded plugins: fastestmirror
 base                                                                                                                                             | 3.6 kB  00:00:00
 updates                                                                                                                                          | 3.4 kB  00:00:00
 Determining fastest mirrors
 .
 .
 .
 .
 Complete!
 Download complete.
 Copy /var/cache/lxc/centos/x86_64/7/rootfs to /var/lib/lxc/centos_lxc/rootfs ...
 Copying rootfs to /var/lib/lxc/centos_lxc/rootfs ...
 sed: can't read /var/lib/lxc/centos_lxc/rootfs/etc/init/tty.conf: No such file or directory
 Storing root password in '/var/lib/lxc/centos_lxc/tmp_root_pass'
 Expiring password for user root.
 passwd: Successsed: can't read /var/lib/lxc/centos_lxc/rootfs/etc/rc.sysinit: No such file or directory
sed: can't read /var/lib/lxc/centos_lxc/rootfs/etc/rc.d/rc.sysinit: No such file or directory

Container rootfs and config have been created.
Edit the config file to check/enable networking setup.

The temporary root password is stored in:

        '/var/lib/lxc/centos_lxc/tmp_root_pass'


The root password is set up as expired and will require it to be changed
at first login, which you should do as soon as possible.  If you lose the
root password or wish to change it without starting the container, you
can change it from the host by running the following command (which will
also reset the expired flag):

        chroot /var/lib/lxc/centos_lxc/rootfs passwd
Please note the above login details, you must require this information for login to the containers.

Containers Credentials:

To log into the container (centos_lxc), either use the temporary root password stored in the following location. In our case “Root-centos_lxc-KRzJLy” is the root password of the centos_lxc.
# cat /var/lib/lxc/centos_lxc/tmp_root_pass

 Root-centos_lxc-KRzJLy
or
Reset the root password using the following command.
# chroot /var/lib/lxc/centos_lxc/rootfs passwd
PS: You are yet to start the containers.

Starting Linux Containers:

After creating the containers, start it using the following command, runs in the background.
# lxc-start -n centos_lxc -d
Now, take the console of the container using the following command.
Note: I use “-t” with “0” to connect the container with tty0, just because tty1 was not responding to me.
# lxc-console -n centos_lxc -t 0
Enter the username and password to login. You can find the credential at the end of the output during the creation of a container. You must the change the root password at the first login.
Sample output:
Connected to tty 0
 Type <Ctrl+a q> to exit the console, <Ctrl+a Ctrl+a> to enter Ctrl+a itself
 CentOS Linux 7 (Core)
 Kernel 3.10.0-327.el7.x86_64 on an x86_64
 centos_lxc login: root
 Password:
 You are required to change your password immediately (root enforced)
 Changing password for root.
 (current) UNIX password:
 New password:
 Retype new password:
 [root@centos_lxc ~]#
Once you logged in, you can perform all the work in this container as like you do in a normal Linux servers.
To exit from the container’s console, press “Ctrl+a” followed by “q”. Now, you will be returned back to host computer’s terminal.
If you want to connect to the container again ( the container is still running), run the following command.
# lxc-console -n centos_lxc -t 0

Working with Linux Containers:

To list down the containers on the host computer, use the following command.
# lxc-ls

 centos_lxc
Also, list down the containers that are currently active and running on the host computer.
# lxc-ls --active

 centos_lxc
Since I have only one container which is currently running that’s why you could see the same output for both of the commands.
If you want to get the full information of the running container, issue the following command.
# lxc-info -n centos_lxc

 Name:           centos_lxc
 State:          RUNNING
 PID:            4047
 IP:             192.168.12.16
 CPU use:        0.47 seconds
 BlkIO use:      6.32 MiB
 Memory use:     4.19 MiB
 KMem use:       0 bytes
 Link:           vethM3N48G
 TX bytes:      1.53 KiB
 RX bytes:      1.94 KiB
 Total bytes:   3.47 KiB
The above command gives you detailed information (name, state, IP address, CPU, memory , I/O and network usage) of “centos_lxc” container.
You can also use IP address to connect to the containers instead of LXC console.
You can stop a running container using “lxc-stop” command, use the following command to stop a “centos_lxc” container.
# lxc-stop -n centos_lxc

Cloning containers:

LXC has a future of cloning a container from the existing container, run the following command to clone an existing “centos_lxc” container to a new container “centos_lxc_clone”.
Note: You must stop a running container before initiating the clone.
# lxc-clone centos_lxc centos_lxc_clone
 Created container centos_lxc_clone as copy of centos_lxc
Check out whether a container is created successfully.
# lxc-ls
 centos_lxc  centos_lxc_clone
You can start working with a new container as usual.
# lxc_start -n centos_lxc_clone -d
# lxc_console -n centos_lxc_clone -t 0
 Connected to tty 0
 Type <Ctrl+a q> to exit the console, <Ctrl+a Ctrl+a> to enter Ctrl+a itself
 CentOS Linux 7 (Core)
 Kernel 3.10.0-327.el7.x86_64 on an x86_64
 centos_lxc_clone login: root
 Password:
 Last login: Sat Mar 26 14:11:17 from 192.168.12.1
 [root@centos_lxc_clone ~]#
Power off the container from inside console.
[root@centos_lxc_clone ~]# poweroff

Taking a Snapshot:

LXC also has an another future called snapshot, use the following commands.
Note: You must stop a container before taking a snapshot.
# lxc-stop -n centos_lxc_clone
For a demo, I am taking the snapshot of centos_lxc_clone.
# lxc-snapshot -n centos_lxc_clone
Sample output:
lxc_container: lxccontainer.c: lxcapi_snapshot: 2879 Snapshot of directory-backed container requested.
lxc_container: lxccontainer.c: lxcapi_snapshot: 2880 Making a copy-clone.  If you do want snapshots, then
lxc_container: lxccontainer.c: lxcapi_snapshot: 2881 please create an aufs or overlayfs clone first, snapshot that
lxc_container: lxccontainer.c: lxcapi_snapshot: 2882 and keep the original container pristine.
To know where the snapshot is being saved, run the following command.
# lxc-snapshot -L -n centos_lxc_clone
 snap0 (/var/lib/lxcsnaps/centos_lxc_clone) 2016:03:26 10:59:10
In Centos 7, LXC snapshots are stored in “/var/lib/lxcsnaps/”

Restoring Snapshot:

To restore a container from the snapshot, use the following command.
# lxc-snapshot -r snap0 -n centos_lxc_clone

Destroying Containers:

To  remove a container completely, use the following command.
# lxc-destroy -n centos_lxc_clone

Running a Ubuntu Container on CentOS 7:

I came across multiple issues when I was trying to run a Ubuntu container on CentOS, was able to successfully run a ubuntu with some tweaks shared in other websites.
Install below packages for Debian based containers.
# yum -y install debootstrap perl
# cd /usr/sbin ; ln -sf debootstrap qemu-debootstrap
Run the following command to replace Debian mirror to ubuntu mirror.
# sed -i 's/DEF_HTTPS_MIRROR="https:\/\/mirrors.kernel.org\/debian"/DEF_HTTPS_MIRROR="https:\/\/mirrors.kernel.org\/ubuntu"/g' /usr/sbin/debootstrap
Get the Ubuntu precise keyring and place it in the keyrings directory.
# cd /tmp
# wget "http://archive.ubuntu.com/ubuntu/pool/main/u/ubuntu-keyring/ubuntu-keyring_2011.11.21.tar.gz"
# tar xzf ubuntu-keyring_2011.11.21.tar.gz
# mkdir /usr/share/keyrings/
# cp /tmp/ubuntu-keyring-2011.11.21/keyrings/ubuntu-archive-keyring.gpg /usr/share/keyrings/
Thanks to blog.toxa.de
Create a Ubuntu container using the following command.
# lxc-create -n ubuntu_lxc -t ubuntu

Checking cache download in /var/cache/lxc/precise/rootfs-amd64 ... 
Installing packages in template: ssh,vim,language-pack-en
Downloading ubuntu precise minimal ...
I: Retrieving Release 
I: Retrieving Release.gpg 
I: Checking Release signature
I: Valid Release signature (key id 630239CC130E1A7FD81A27B140976EAF437D05B5)
I: Retrieving Packages 
I: Validating Packages 
I: Retrieving Packages 

Generation complete.
Setting up libdevmapper1.02.1 (2:1.02.48-4ubuntu7.4) ...
Setting up dmsetup (2:1.02.48-4ubuntu7.4) ...
update-initramfs: deferring update (trigger activated)
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
Processing triggers for initramfs-tools ...
Processing triggers for resolvconf ...
invoke-rc.d: policy-rc.d denied execution of start.
Download complete
Copy /var/cache/lxc/precise/rootfs-amd64 to /var/lib/lxc/ubuntu_lxc/rootfs ... 
Copying rootfs to /var/lib/lxc/ubuntu_lxc/rootfs ...
Generating locales...
  en_US.UTF-8... up-to-date
Generation complete.
Creating SSH2 RSA key; this may take some time ...
Creating SSH2 DSA key; this may take some time ...
Creating SSH2 ECDSA key; this may take some time ...
Timezone in container is not configured. Adjust it manually.
##
# The default user is 'ubuntu' with password 'ubuntu'!
# Use the 'sudo' command to run tasks as root in the container.
##
From the above output, you can see that default user is “ubuntu” and the password for the user is “ubuntu”
Take a console of Ubuntu container with the following command.
# lxc_console -n ubuntu_lxc

Ubuntu 12.04.5 LTS ubuntu_lxc console
ubuntu_lxc login: ubuntu
Password: 
Welcome to Ubuntu 12.04.5 LTS (GNU/Linux 3.10.0-327.10.1.el7.x86_64 x86_64)
 * Documentation:  https://help.ubuntu.com/
The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.
Would you like to enter a security context? [N]  
ubuntu@ubuntu_lxc:~$
PS: I have not checked the functionality of the server except login, please do post here for any issues.
That’s All! , thanks for reading. Be social, share this with your friends.

Linux Performance Metrics & Monitoring

Rule of Thumb Metrics would be the following:

CPU Utilization

CPU usage is usually the first place we look when a server shows signs of slowing down (although more often than not, the problem is elsewhere).  The top command is arguably the most common performance-related utility in Linux when it comes to processes and CPU.  By default, top displays summary percentages for all CPUs on the system.   It is important to distinguish between two types of CPU metrics: load averages and percentages.


Load Averages

All UNIX-like systems traditionally display the CPU load as 1-minute, 5-minute and 15-minute load averages.  Essentially, the load average represents the fraction of time that the CPU is busy.  Remember that a CPU can be over-utilized – processes can be waiting for a CPU to become available, so you could see utilization rates over 1.00.  The “perfect” utilization point of 1.00 per CPU means that CPU is executing 100% of the time and no processes are waiting for a CPU to become available.  (On a machine with a single dual-core CPU that point would be 2.00, on a dual quad-core CPU – 8.00 and so on).  Of course a utilization of 1.00 per CPU would mean that there is no spare capacity to take an increased load, so most administrators are worried when they see utilization numbers consistently over 0.70.  Another command which displays CPU percentage statistics is mpstat.


Memory Usage

When a process requests the kernel to allocate memory and the system has run out of physical memory, the kernel will start paging out the least-used memory blocks to disk to free up some space, until the process that allocated them needs them back, at which point the kernel will have to find another least-used block, page it out and page in the original block in physical memory.  This mechanism means that more memory is available to applications than the physical memory installed on the server – this memory is known as virtual memory.  The good thing is that your application doesn’t even know it is using virtual memory.  But that doesn’t mean you should not keep track of memory usage because nothing is free.  Since disk access is slower than RAM access, if  your system starts paging excessively, virtual memory access will become a performance bottleneck. (A quick note: although the terms paging and swapping are often used interchangeably, strictly speaking paging refers to individual memory pages being loaded or saved to disk, and swapping – to the entire memory space of a Linux process being moved from memory to disk or vice versa).   To examine the virtual memory usage, use the vmstat command. When run without parameters, it displays a snapshot of the current state of virtual memory.  If you discover that certain applications need too much memory and your system is paging more than it should, consider installing more memory or moving them to another machine. If you do install more memory, don’t forget to increase the amount of swap space – as a rule of thumb it should be at least equal to the amount of physical memory available.


Disk Subsystems

Whenever you suspect that disk I/O activity is the bottleneck, use the iostat command with the -x switch to examine disk activity.  For obvious reasons, the I/O subsystem is the most common bottleneck on database servers and file servers. 

Some options to consider are:
  • Use faster disks. Higher RPM means faster seek time
  • Use logical volumes with striping – this way a single request can be serviced by several disks in parallel
  • Use a hardware RAID controller – avoid software RAID for data-intensive applications
  • Add more memory to allow for larger buffers