Cross compiling Hello World

Let's start with a simple example, your typical Hello World first application. To compile it natively you'll simply have to do:

$ gcc hello.c -o hello

If you want to cross compile it you first need to have a cross toolchain. Nowadays most distributions include a pretty good one, so getting it is just one command away. On Debian for example:

$ sudo apt-get install gcc-5-arm-linux-gnueabihf

Since your Hello World is a simple application with few dependencies (and supposing that the cross toolchain is correctly configured), cross compiling should be no more complicated than:

$ arm-linux-gnueabihf-gcc-5 hello.c -o hello-arm

Using file and ldd commands you can easily verify that you have an ARM executable:

$ file hello
        hello: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=a8ab6d12595b904a994f769eb8cce0a3ae02ffad, not stripped
$ file hello-arm
        hello-arm: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.32, BuildID[sha1]=0c4e83107a35a9331cbd0e84bfaf28ae1355366c, not stripped
$ ldd hello
        linux-vdso.so.1 (0x00007fff6ade4000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007efeefbf8000)
        /lib64/ld-linux-x86-64.so.2 (0x00007efeeffa1000)
$ ldd hello-arm
        not a dynamic executable

Cross compiling a real application

Now we are going to try to do the same thing but with a real, and very useful, application: sl. The code of this application is simple and short but it has real dependencies other than the libc.

First, natively:

$ git clone https://github.com/mtoyoda/sl.git
$ cd sl
$ make
        gcc -O -o sl sl.c -lncurses
        sl.c:39:20: fatal error: curses.h: No such file or directory
        #include <curses.h>
        compilation terminated.
        Makefile:13: recipe for target 'sl' failed
        make: *** [sl] Error 1

If you are used to building software it should be pretty obvious for you here: you're missing a dependency and have to install the -dev version of the required library:

$ sudo apt-get install libncurses5-dev libncursesw5-dev
$ make

Bravo! We managed to build our application natively… Now let's try to cross compile it. First we need to replace CC=gcc by CC=arm-linux-gnueabihf-gcc-5 in the Makefile (yes, this is not very cross compile friendly) and then try to cross compile:

$ make
        arm-linux-gnueabihf-gcc-5 -O -o sl sl.c -lncurses
        /usr/lib/gcc-cross/arm-linux-gnueabihf/5/../../../../arm-linux-gnueabihf/bin/ld: cannot find -lncurses
        collect2: error: ld returned 1 exit status
        Makefile:13: recipe for target 'sl' failed
        make: *** [sl] Error 1

This time curses.h exists on our system but it can't find the ARM version of the lib to link against. We installed the amd64 version of the lib earlier but how to install the ARM version and the linker to find it?

That part has been a pain for a lot of people for a long time. You can even find tutorials on the Internet of people advising you to install the libraries on your Raspberry Pi, mount your Raspberry on your system and modify the sysroot to change where to look for libs. Please, have some mercy, don't do that.

Fortunately some Linux distributions (and particularly Debian) have made a lot of progress in supporting Multiarch. Here is the correct way to add support for ARM and install the ARM version of the libs to a Debian-like system:

$ sudo dpkg --add-architecture armhf
$ sudo apt-get update
$ sudo apt-get install libncurses5-dev:armhf libncursesw5-dev:armhf
$ make

Done!

Cross compiling a bigger software

For the moment we only cross compiled small and simple software, now we have to try something more complex and with autotools. Using a really advanced software picking algorithm, I chose e2fsprogs.

Let's skip the native compilation, a simple ./configure && make works great as usual. So let's go directly to the cross compilation:

$ git clone git://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git
$ cd e2fsprogs/
$ ./configure --host arm-linux-gnueabihf #  We indicate to ./configure on which arch this is going to run
$ make
        ../../lib/libuuid.a: error adding symbols: Archive has no index; run ranlib to add one
        collect2: error: ld returned 1 exit status
        Makefile:411: recipe for target 'tst_uuid' failed
        make[2]: *** [tst_uuid] Error 1

And here we're stuck… This error is not as classical and trivial as the previous ones (no, Stackoverflow will not give you a direct command to solve it) and it's not the point of this article to solve it. Clearly something is missing or not set properly on our system. Solving it could take us anywhere from 30 seconds to a day or two, and nothing says that there is no other problem that hiding behind.

Using schroot to cross compile

Cross compiling can be tricky and painful, what we would like to do is compiling natively as usual. The obvious solutions would be to tranfer your sources on your ARM board and compile on it but that will probably be very slow, not easily automated, hard to use in a continuous integration workflow (but not impossible with projects like LAVA) and just not a nice process.

What we really want is to be able to do native compilation on our PC or any computer/server. To do so we are going to take advantage of QEMU to emulate an ARM platform. We will run a native ARM gcc on it, use debootstrap to create a full ARM compatible system around the toolchain and schroot to ease their use.

$ sudo apt-get install debootstrap schroot sbuild qemu-user-static

We start by creating our ARM system with debootstrap:

$ sudo debootstrap --variant=default --arch=armhf --foreign jessie /tmp/sbuild-jessie-armhf http://debian.mirrors.ovh.net/debian
$ sudo cp /usr/bin/qemu-arm-static /tmp/sbuild-jessie-armhf/usr/bin/
$ sudo chroot /tmp/sbuild-jessie-armhf/ ./debootstrap/debootstrap --second-stage

At this step we could use this system with chroot (and in fact we did to finalize the debootstrap) and use it to emulate an ARM platform. Instead we are going to transform it into a schroot (which means secure chroot) because it provides a lot of improvements and features we need compared to a regular chroot.

• Usable with non-root users
• Permissions checking
• Automated mounting of /home /proc /dev /sys /tmp
• Automated copying of network files: resolv.conf, networks, hosts, protocols
• Automated copying of users files: passwd, shadow, group
• Clean and reproducible environment
• (Possibility to automatically build Debian packages with sbuild)

What you have to understand here is that when you will enter the schroot you will have everything you have in your regular environment, your users, sudoers, access to network, to your devices, to your home, etc.

You will also note that we're creating our schroot in the form of a tarball. This is not mandatory but personnally I find this way cleaner. When you use your schroot it will be untared and when you're done it will simply be dropped. The source of your schroot will not be impacted by whatever you have done while chrooted.

$ sudo sbuild-createchroot --arch=armhf --foreign --setup-only --make-sbuild-tarball=/var/lib/sbuild/build/jessie-armhf.tar.gz jessie /tmp/sbuild-jessie-armhf/ http://debian.mirrors.ovh.net/debian
$ sudo sbuild-adduser $USER

That's it! Your ARM schroot is created, you just need to log-off and on again to be able to use it. You could now simply enter it with the command schroot -c jessie-armhf-sbuild, do whatever you want and quit with exit. Remember, whatever you will do in the system will be lost when you exit the schroot but you may want to install some tools or update the schroot. To do so you will need these specific commands:

$ sudo sbuild-update -udcar jessie-armhf-sbuild #  update, upgrade, dist-upgrade, clean, etc.
$ sudo sbuild-apt jessie-armhf-sbuild -- apt-get -y install build-essential #  Permanently install some useful tools for what we want to do…

Now that we really are ready to use our schroot to build software, let's get back to e2fsprogs.

We could, again use a direct call to our schroot, but here we're going to use a session. The session is a feature that allows you to untar your schroot (which avoids to wait a couple of seconds everytime), keep it open in background as long as you need and close it when you decide to.

$ export SESSION_CHROOT=$(schroot -b -c jessie-armhf-sbuild) #  Untar a schroot session and keep its name

Now that the session is open we can use it as much as we want. Since /home is mounted by schroot, if you are in your e2fsprogs/ folder you will stay in it when you enter the schroot and to build e2fsprogs you can do it exactly as we did for the native compilation:

$ schroot -r -c $SESSION_CHROOT -- ./configure
$ schroot -r -c $SESSION_CHROOT -- make

Yes, nothing more, it's done, your software is built for ARM and it has directly been built in your e2fsprogs/ folder, you don't need to move files from and to your chroot.

And when you are done with the schroot, don't forget to close your session:

$ schroot -e -c $SESSION_CHROOT

Another command that can be useful to list all schroots:

$ schroot -l --all
        chroot:jessie-armhf-sbuild
        session:jessie-armhf-sbuild-d34ba516-4281-4b5c-a3cb-9e60112ea2ea
        source:jessie-armhf-sbuild

You can note that even if we only have one schroot the command is listing 3 of them that in fact are the same.

• chroot: designates the default target (you don't need to specify it), it's your regular schroot.
• source: designates the source of your schroot, if you specify it, all modifications you could do (like installing software) will be saved back in the tar of your schroot when you leave it (it's another option to the commands sbuild-update and sbuild-apt used previously).
• session: designates your open sessions.

Comparisons

As we saw, cross compiling through schroot is easy but it has one big drawback… It's terribly slow.

Depending on the size of your software, the number of dependencies, the machine you build it on and the time you can wait for a build it's up to you to chose between a real cross compiler and a schroot.

For example, the kernel is a big software that has almost no dependencies… you'd be better using a cross compiler. But your own custom app will probably be a lot smaller and faster to compile with a schroot whitout any hassle of cross compilation…