When starting out with Embedded Linux, it is easy to assume that since a C++ application works on our computer, simply copying it to the device will suffice. In many IoT projects, however, we write code in a different environment than the one in which the program will later run. The reality often looks quite different.
Will Code Written on a Computer Work on the Board?
For example, we might work on an x86_64 computer while the target IoT gateway runs on ARM or ARM64. The source code may be identical, but the compiled program contains instructions for a specific processor architecture. Therefore, a binary built with standard g++ on x86_64 will not simply work on an ARM device.
In such cases, we need cross-compilation of C++ applications: building an app on one platform with the intention of running it on another. Simplified, the flow of such a process looks like this:
x86_64 computer → CMake → cross-toolchain → ARM application → Linux IoT device
Host and Target
In the context of cross-compilation of C++ applications, the host is the computer where we perform the build—for example, an Ubuntu laptop on x86_64. The target, on the other hand, is the device where we will later run the application.
In an IoT product, this could be a gateway or an edge device based on ARM and Embedded Linux. Both terms appear frequently when discussing cross-compilation.
When we build an application on our computer to run locally, the host and target are usually the same platform. With cross-compilation, they are different platforms, so during the build we need elements from both environments. CMake and some of the tools run on the host, while the libraries we link against should correspond to the target. This complicates configuration a bit.
What Is a Toolchain?
The most visible element of cross-compilation is the cross-compiler. Instead of a compiler for our host machine, we use one that generates code for ARM. Together with the linker and other tools needed to build the application, it forms a toolchain. However, the cross-compiler itself is just the beginning. A toolchain is a collection of interacting tools that process written code step-by-step into a format that ARM understands.
In a CMake project, we can store information about the target environment in a toolchain file. The official CMake documentation for cross-compilation under Linux shows settings such as the target system, architecture, C/C++ compilers, and an optional sysroot. A simple file might look like this:
set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_PROCESSOR aarch64)
set(CMAKE_C_COMPILER aarch64-linux-gnu-gcc)
set(CMAKE_CXX_COMPILER aarch64-linux-gnu-g++)
set(CMAKE_SYSROOT /path/to/sysroot)
When configuring the project, we specify to CMake which toolchain file to use. As a result, we don’t have to pass all platform details manually every time. CMake knows right from the start that we are building the project for an environment different from the one it is running on.
A Cross-Compiler Is Not Enough
For a very simple application, a suitable aarch64-linux-gnu-g++ might be enough. The problem arises, however, when the project relies on additional libraries.
Suppose the application uses a library to communicate with hardware. On our laptop, we have its headers and the .so file, so a standard build works. The target device might have a different version of this library, or a version built for ARM instead of x86_64. Or it might not have the library at all.
When cross-compiling, we need headers and libraries that correspond to the target. Care must be taken not to accidentally use libraries from our laptop /usr/lib.
Sysroot as the Target Environment
A sysroot can be thought of as a directory containing the files required to build applications for the target. It includes, among other things, headers and libraries prepared for the target platform.
Yocto documentation describes sysroot as a structure mimicking the target’s filesystem, used during cross-compilation, among other things. The target sysroot contains the libraries and headers needed to build code for the device. For an application built on x86_64 and run on ARM, this can be simplified to:
host: x86_64 Linux
toolchain: aarch64-linux-gnu-g++
sysroot: libraries + headers for target
target: ARM Embedded Linux
The compiler runs on our computer, but it uses components corresponding to the device’s environment.
CMake Configuration
A sysroot alone does not solve every problem. CMake often searches for libraries using find_package(), find_library(), or find_path(). While convenient for native builds, during cross-compilation you must make sure it didn’t pick up host files when it should have been searching for target equivalents.
We can control this behavior, for example, like this:
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)
Programs executed during the build should usually come from the host. Libraries and headers used to build the application for the device should come from the target environment. Without this separation, a build might appear successful while still relying on wrong dependencies.
Deploying the Build Is Not Everything
A successful build does not always mean the application will work correctly on the device. We might produce an ARM binary, only to copy it to the target and encounter an error about a missing library. The program might also expect a different version of a .so file than what is available on the system.
A harder case to spot is when CMake finds a host dependency instead of one from the sysroot during configuration. With many dependencies, this is easy to overlook. In cross-compilation, the build environment is part of the project itself. Therefore, when troubleshooting, besides inspecting the code, it is worth checking:
which compiler CMake is using,
which architecture the binary was created for,
which sysroot is being used,
where CMake found the libraries,
which libraries are actually available on the device.
Where Does IoT Fit In?
Cross-compilation is not unique to IoT, but in Linux IoT devices products, it is a common real-world scenario. For example, a gateway collects data from several local devices via BLE, UART, or CAN. It performs part of the processing locally and sends the results to a server.
Such a gateway runs on an ARM module with Embedded Linux, while we develop the C++ application on a significantly more powerful x86_64 computer. Sometimes you can compile directly on the device, especially during prototyping. In a larger project, however, it is much more convenient to have a controlled environment and know exactly which target you are building for.
This also applies to CI. The exact same build should be reproducible without having a physical device standing next to the developer’s computer.
Yocto and Ready-to-Use SDKs
If the device runs an OS built with Yocto, an SDK generated along with the system image can solve some of these problems. A Yocto SDK includes, among other things, a cross-development toolchain as well as libraries and headers tailored to the specific target environment. This reduces the risk of a developer manually pairing a toolchain from one system version with libraries from another.
Choosing Yocto over a traditional Linux distribution involves other trade-offs as well. We covered them in more detail in our article Yocto vs Debian for Embedded Systems Development. Yocto can be particularly useful when you need deep control over image content and a reproducible build environment.
We’ve also featured Yocto in previous projects. An example is a system built on a Toradex i.MX7 module, where one task was optimizing Embedded Linux boot time. You can read more about this project in the article Embedded Linux boot time optimization for a custom application. This example demonstrates that the target environment matters much earlier than when simply copying the finished application to the device.
Common Pitfalls
A large share of cross-compilation issues do not stem from C++ code errors. They often need to be investigated within the build and target environments. For example, it might turn out that:
you are using a compiler for the wrong architecture,
the sysroot comes from a different system version than the one on the device,
CMake finds host libraries instead of target ones,
the required library is missing from the device’s root filesystem,
the library version on the device differs from the one used during linking,
the configuration works locally, but CI has different paths or dependencies.
In such cases, compiler messages alone are often not enough. Start by verifying that you are building the application for the correct target device.
Testing on a Real Device Is Still Necessary
Cross-compilation allows you to prepare an application without building it directly on the device. However, running and testing it on the target is still necessary. Verifying the build configuration, dependencies, and file architecture doesn’t replace this.
The device may have different memory limits, access to specific hardware interfaces, or its own custom system configuration. Therefore, the process should also include deploying and verifying the program on the device:
code → cross-build → deploy → execution on target → test
We shouldn’t stop at just the build stage:
code → build → assume it works
Key Takeaways
Cross-compilation of C++ applications allows building an application on one platform (e.g., x86_64) for another target (e.g., ARM).
A CMake toolchain file specifies the target platform and build tools.
A sysroot provides target headers and libraries. A cross-compiler alone is often not enough.
A successful build does not guarantee that the application will run on the device. You must check the architecture, dependencies, and target environment.
In systems created with Yocto, an SDK can provide a cross-toolchain and sysroot tailored to a specific system image.
