Aug 13, 2013

Install Debian GNU/Linux on a FreeBSD Jail with ZFS

Continuing with the process of configuring a  newly installed FreeBSD system to get a production environment with hosted services in Jails. 

We will continue working with Debian and Linux-based distributions that provide the right Userland to meet the goal, not fully achieved, of executing  Linux binaries on FreeBSD Jail.


Preliminary considerations

Binary compatibility for Linux that provides FreeBSD 9 is limited to 32-bit applications. FreeBSD provides binary interface, not a 'emulation', with support for Linux kernel up to version 2.6.16. So when selecting a version of Debian Linux should keep in mind these limitations.

 

Creating our Jail

The implementation of a Debian GNU/Linux Jail is similar to a Debian GNU/kFreeBSD Jail, however, has significant differences which must be highlighted.

1. - Install the tool Debootstrap, which installs Debian in a subdirectory of another system.
root@morsa:/root # portsnap fetch update 
root@morsa:/root # portmaster sysutils/debootstrap
2. - Configure the file system where we will install the Jail.
root@morsa:/root # zfs create -o mountpoint=/jailz fbsdzpool1/jailz
root@morsa:/root # zfs create fbsdzpool1/jailz/deb-master
root@morsa:/root # mkdir /jailz/etc
3 - Load the necessary modules for Linux compatibility.
 root@morsa:/root # kldload fdescfs linprocfs linsysfs tmpfs
4.- We enable Jails support and compatibility with Linux.
root@morsa:/root # echo 'jail_enable="YES"' >> /etc/rc.conf
root@morsa:/root # echo 'linux_enable="YES"' >> /etc/rc.conf
5.- Only if installing squeeze, change the compatibility level declared 2.6.18.
root@morsa:/root #  echo 'compat.linux.osrelease=2.6.18' >> /etc/sysctl.conf
6.- Include in the configuration file /jailz/etc/jail.conf:

// Start fragment
deb-master {
  path = /jailz/deb-master;
  allow.mount;
  mount.devfs;
  host.hostname = deb-master;
  mount.fstab="/jailz/etc/fstab.deb-master";
  ip4.addr = 127.0.0.10;
  interface = lo0;
  exec.start = "/etc/init.d/rc 3";
  exec.stop = "/etc/init.d/rc 0";
}
// End of excerpt

7.- Define the mounting points for the Jail in  /jail/etc/fstab.dev-master: 

// Begin the File /jailz/etc/fstab.deb-master
linsys   /jailz/deb-master/sys         linsysfs  rw          0 0
linproc  /jailz/deb-master/proc        linprocfs rw          0 0
tmpfs    /jailz/deb-master/lib/init/rw tmpfs     rw,mode=777 0 0

// End of the File /jailz/etc/fstab.deb-master


8.- With debootstrap install Debian GNU/Linux, lenny or squeeze versions, in the path of the Jail.
root@morsa:/root # debootstrap --foreign --arch=i386 lenny /jailz/deb-master http://archive.debian.org/debian/
I: Retrieving Release
W: Cannot check Release signature; keyring file not available /usr/share/keyring
s/debian-archive-keyring.gpg
I: Retrieving Packages
I: Validating Packages
I: Resolving dependencies of required packages...
I: Resolving dependencies of base packages...
I: Checking component main on http://archive.debian.org/debian...
I: Retrieving libacl1 2.2.47-2
I: Validating libacl1 2.2.47-2
I: Retrieving adduser 3.110
I: Validating adduser 3.110
I: Retrieving apt-utils 0.7.20.2+lenny2
I: Validating apt-utils 0.7.20.2+lenny2
I: Retrieving apt 0.7.20.2+lenny2
....
I: Extracting login...
I: Extracting passwd...
I: Extracting libslang2...
I: Extracting initscripts...
I: Extracting sysv-rc...
I: Extracting sysvinit-utils...
I: Extracting sysvinit...
I: Extracting tar...
I: Extracting tzdata...
I: Extracting bsdutils...
I: Extracting mount...
I: Extracting util-linux...
I: Extracting zlib1g...
9. - Inside the Jail, delete  the configuration files sysvinit_*.
root@morsa:/jailz/etc # ls /jailz/deb-master/var/cache/apt/archives/sysvinit_*
/jailz/deb-master/var/cache/apt/archives/sysvinit_2.86.ds1-61_i386.deb
root@morsa:/jailz/etc # rm /jailz/deb-master/var/cache/apt/archives/sysvinit_*
  10. - We mount the filesystems of the Jail, in this case, takes place after installation with debootstrap.
root@morsa:/root # mount -t linprocfs none /jailz/deb-master/proc
root@morsa:/root # mount -t devfs none /jailz/deb-master/dev
root@morsa:/root # mount -t linsysfs none /jailz/deb-master/sys
root@morsa:/root # mount -t tmpfs none /jailz/deb-master/lib/init/rw 
 11. - Run a shell with chroot within the path of the Jail.
root@morsa:/root # chroot /jailz/deb-master /bin/bash
12. - To complete the configuration of the Jail, from the shell started in the chroot environment, run :
 I have no name!@morsa:/# dpkg --force-depends -Ei /var/cache/apt/archives/*.deb
13. -  We left the previous shell and unmount the previously mounted file systems in step 10.
root@morsa:/root # umount /jailz/deb-master/proc
root@morsa:/root # umount /jailz/deb-master/dev
root@morsa:/root # umount /jailz/deb-master/sys
root@morsa:/root # umount /jailz/deb-master/lib/init/rw
 In the likely event that can not be unmounted /jailz/deb-master/dev because the filesystem is busy, run:
# fstat | grep deb-master
# kill -9 PID (For each process listed in the previous step)
Or, restart the system.

Never start the Jail without cleaning the processes and unmounting /jailz/deb-master/dev.

14. - You need to disable rsyslog inside the Jail,  because is not supported by the Linux compatibility module.

Therefore, before starting the Jail, for each directory in the path /jailz/deb-master/etc/rcX.d (where X takes values from 0 to 6) rename the service startup scripts.
# mv S10rsyslog _S10rsyslog
# mv K90rsyslog _K90rsyslog
15. - Start the Jail, check that is correctly started and login.
root@morsa:/root # jail -f /jailz/etc/jail.conf -c deb-master
deb-master: created
Starting periodic command scheduler: crond.

root@morsa:/root # jls
JID IP Address Hostname Path
1 127.0.0.10 deb-master /jailz/deb-master

root@morsa:/root # jexec 1 /bin/bash
deb-master:/# uname -a
Linux deb-master 2.6.16 FreeBSD 9.1-RELEASE-p4 #0: Mon Jun 17 11:42:37 UTC 2013 i686 GNU/Linux
16. - Edit the file /etc/apt/sources.list and correct their content.
// Start the excerpt.
http://archive.debian.org/debian/ deb lenny main contrib non-free
// End of excerpt

17. - Update the package list.
deb-master:/# apt-get update
Get:1 http://archive.debian.org lenny Release.gpg [1034B]
Get:2 http://archive.debian.org lenny Release [99.6kB]
Get:3 http://archive.debian.org lenny/main Packages [6872kB]
Get:4 http://archive.debian.org lenny/non-free Packages [124kB]
Get:5 http://archive.debian.org lenny/contrib Packages [94.3kB]
Fetched 7191kB in 11s (649kB/s)
Reading package lists... Done
18. - Shutdown the Jail, the error messages are due to processes within the Jail can not perform certain operations.
root@morsa:/root # jail -f /jailz/etc/jail.conf -r deb-master
umount2: Operation not permitted
umount: fbsdzpool1/jailz: must be superuser to umount
umount2: Operation not permitted
umount: fbsdzpool1/ROOT/91_30062013/usr: must be superuser to umount
umount2: Operation not permitted
umount: fbsdzpool1/ROOT/91_30062013/usr: must be superuser to umount
umount2: Operation not permitted
umount: fbsdzpool1/ROOT/91_30062013/var: must be superuser to umount
umount2: Operation not permitted
umount: fbsdzpool1/ROOT/91_30062013/var: must be superuser to umount
failed.
mount: fbsdzpool1/ROOT/91_30062013: unknown device
Will now halt.
ifdown: shutdown usbus0: Invalid argument
ifdown: shutdown ath0: Invalid argument
ifdown: shutdown usbus1: Invalid argument
ifdown: shutdown lo0: Invalid argument
ifdown: shutdown lo0: Invalid argument
ifdown: shutdown eth1: Invalid argument
deb-master: removed
 19. - Make a ZFS snapshot of the Jail.
root@morsa:/root # zfs snapshot zfs snapshot fbsdzpool1/jailz/deb-master@lenny 
Now, we have a Base Jail, from which we can generate new Jails with ZFS clones, then  we added a new entry to /jailz/jail.conf and we created the file /jailz/etc/fstab.newjail.
# zfs clone fbsdzpool1/jailz/deb-master@lenny fbsdzpool1/jailz/newjail
On these cloned Jails, we can begin to run 32-bit linux apps with their own Linux elf format.

 

Final Thoughts

All final thoughts about to install Debian GNU/kFreeBSD in a Jail are equally applicable to this post.

In this post I installed Userlands with versions of lenny and squeeze, but I recommend using the lenny release. The Linux  compatibility that FreeBSD provides is for kernel version 2.6.16 and minimal kernel version required by squeeze is 2.6.18;  although it works fairly well, those applications using features specific to the 2.6.18 not work at all.

As a final note, to enable internet access from the Jail, required to install packages, we enable the pf module, then we make NAT with the IP addresses of the Jails, to do this execute the following:

root@morsa:/root # kldload pf
root@morsa:/root # echo 'pf_enable="YES"' >> /etc/rc.conf
root@morsa:/root # echo "nat on wlan0 from 127.0.0.10 to any -> (wlan0)" >> /etc/pf.conf

7 comments:

  1. Excellent and very useful !

    zfs snapshot zfs snapshot fbsdzpool1/jailz/deb-master@lenny <-- here you have put one 'zfs snapshot' too many :)

    ReplyDelete
  2. Very interesting and complete HOWTO! Thanks a lot!
    But you have an error on step 16, the file must be:
    deb http://archive.debian.org/debian/ lenny main contrib non-free

    ReplyDelete
  3. There must be linux jail snapshots which can be imported!

    ReplyDelete
  4. Nice start. But, a lot of people, myself included, are looking to dump other problematic operating systems and yet still require a full desktop environment. You've stopped at networking (no wifi mentioned) and say nothing of running xwindows. Is it simply too difficult to set up a DE/WM environment in a bsd jail?

    ReplyDelete
    Replies
    1. This setup is intended for server environments, not for desktops.

      Delete

Comments are welcome, I encourage you to contribute by proposing topics of your interest to develop in this blog.