Clang cross compile windows linux

Время на прочтение21 мин

Количество просмотров27K

Каждый, кто когда-либо пробовал собрать программу на C/C++ через кросс-компиляцию знает, насколько болезненным может быть этот процесс. Главными причинами столь печального положения вещей являются недружелюбность систем сборки при конфигурации кросс-компиляции, а также запутанность процесса настройки набора утилит (тулчейна).

Одним из основных виновников этих проблем, по моему опыту, является тулчейн GNU — древний мамонт, на котором много десятилетий строится весь мир POSIX. Подобно многим компиляторам былых времён, семейство GCC и binutils никогда не ориентировалось на поддержку множества различных целей сборки в одной установке, и единственным способом хоть как-то добиться желаемого была настройка полной кросс-билд-системы для каждой целевой платформы на каждом хосте.

Например, если вы хотите собрать что-то для FreeBSD на машине под Linux с помощью GCC, вам потребуется:

  • Установленный GCC + binutils для вашего сборочной платформы (т.е. x86_64-pc-linux-gnu или подобное);
  • Полностью установленный GCC + binutils для вашей целевой платформы (т.е. x86_64-unknown-freebsd12.2-gcc, as, nm и т.д.)
  • Sysroot со всеми необходимыми библиотеками и заголовочными файлами, который вы можете собрать самостоятельно, либо утащить из работающей FreeBSD.

Некоторые дистрибутивы Linux, а также некоторые разработчики оборудования могут облегчить этот процесс, предоставляя готовые наборы для компиляции, однако их никогда не хватает, ввиду огромного количества различных комбинаций сборочных/целевых платформ. Зачастую это означает, что вам необходимо самостоятельно полностью собрать тулчейн с нуля, что приводит к масштабным затратам времени и ресурсов, если конечно у вас не самый мощный процессор.

Clang как кросс-компилятор

Эти досадные ограничения побудили меня обратить внимание на LLVM (и Clang), который изначально создан как полноценный тулчейн для кросс-компиляции, и при этом практически полностью совместим с GNU. Один единственный инстанс LLVM способен собирать и компилировать код для каждой поддерживаемой платформы; помимо него для сборки нужен лишь sysroot.

Несмотря на то, что он ещё не может сравниться по удобству с тулчейнами современных языков (такими, как gc и GOARCH/GOOS в Go), это всё же настоящий глоток свежего воздуха по сравнению со сложностями настроек тулчейнов GNU. Вы можете просто установить его из пакетов для вашего дистрибутива (если только он не сильно старый), и сразу избежать всех сложностей с множественными установками GCC.

Ещё несколько лет назад весь процесс не был настолько хорошо настроен. Поскольку LLVM ещё не включал в себя весь тулчейн, вам нужно было по-прежнему откуда-то брать binutils, специфичный для вашей целевой платформы. И даже хотя решить эту проблему было гораздо проще, чем собрать весь компилятор целиком (binutils собирается значительно быстрее), этот факт всё же добавлял хлопот. Однако к настоящему моменту llvm-mc (интегрированный ассемблер LLVM) и lld (универсальный линкер) уже стабильны и настолько же гибки, как весь остальной LLVM.

Когда весь тулчейн доступен, для компиляции и сборки вашего проекта необходим ещё sysroot, содержащий все необходимые библиотеки и заголовочные файлы.

Добываем sysroot

Быстрее всего раздобыть работающую системную папку для нужной ОС можно, скопировав её напрямую из существующей системы (для этого зачастую подходит контейнер Docker). Например, вот так я скопировал работающий sysroot из виртуалки с FreeBSD 13-CURRENT AArch64 с помощью tar и ssh

$ mkdir ~/farm_tree
$ ssh FARM64 'tar cf - /lib /usr/include /usr/lib /usr/local/lib /usr/local/include' | bsdtar xvf - -C $HOME/farm_tree/

Примечание

При копировании по сети, а не локально, неплохо также сжать получившийся tarball

Запускаем кросс-компилятор

Когда всё готово, остаётся лишь запустить Clang с правильными аргументами:

$  clang++ --target=aarch64-pc-freebsd --sysroot=$HOME/farm_tree -fuse-ld=lld -stdlib=libc++ -o zpipe zpipe.cc -lz --verbose
clang version 11.0.1
Target: aarch64-pc-freebsd
Thread model: posix
InstalledDir: /usr/bin
 "/usr/bin/clang-11" -cc1 -triple aarch64-pc-freebsd -emit-obj -mrelax-all -disable-free -disable-llvm-verifier -discard-value-names -main-file-name zpipe.cc -mrelocation-model static -mframe-pointer=non-leaf -fno-rounding-math -mconstructor-aliases -munwind-tables -fno-use-init-array -target-cpu generic -target-feature +neon -target-abi aapcs -fallow-half-arguments-and-returns -fno-split-dwarf-inlining -debugger-tuning=gdb -v -resource-dir /usr/lib/clang/11.0.1 -isysroot /home/marco/farm_tree -internal-isystem /home/marco/farm_tree/usr/include/c++/v1 -fdeprecated-macro -fdebug-compilation-dir /home/marco/dummies/cxx -ferror-limit 19 -fno-signed-char -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -faddrsig -o /tmp/zpipe-54f1b1.o -x c++ zpipe.cc
clang -cc1 version 11.0.1 based upon LLVM 11.0.1 default target x86_64-pc-linux-gnu
#include "..." search starts here:
#include <...> search starts here:
 /home/marco/farm_tree/usr/include/c++/v1
 /usr/lib/clang/11.0.1/include
 /home/marco/farm_tree/usr/include
End of search list.
 "/usr/bin/ld.lld" --sysroot=/home/marco/farm_tree --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --enable-new-dtags -o zpipe /home/marco/farm_tree/usr/lib/crt1.o /home/marco/farm_tree/usr/lib/crti.o /home/marco/farm_tree/usr/lib/crtbegin.o -L/home/marco/farm_tree/usr/lib /tmp/zpipe-54f1b1.o -lz -lc++ -lm -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /home/marco/farm_tree/usr/lib/crtend.o /home/marco/farm_tree/usr/lib/crtn.o
$ file zpipe
zpipe: ELF 64-bit LSB executable, ARM aarch64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 13.0 (1300136), FreeBSD-style, with debug_info, not stripped

В приведённом примере я собрал исходник C++ в программу для платформы AArch64 FreeBSD, и всё это с использованием лишь clang и lld, которые я уже установил на своей машине GNU/Linux.

Чуть-чуть деталей:

  1. --target переключает целевую платформу LLVM по умолчанию (x86_64-pc-linux-gnu) в aarch64-pc-freebsd, таким образом включая кросс-компиляцию.
  2. --sysroot заставляет Clang использовать указанный путь как корневой для поиска библиотек и заголовочных файлов, вместо обычных путей. Заметьте, что иногда этого ключа недостаточно, особенно если проект использует GCC, а Clang не может определить путь к нему. Такая проблема может быть легко исправлена указанием ключа --gcc-toolchain, который указывает, где найти установленный GCC.
  3. -fuse-ld=lld указывает Clang использовать lld вместо любого другого линковщика, используемого в системе. Скорее всего системный линковщик, доступный по умолчанию, не сможет собрать «чужую» программу, в то время как LLD поддерживает напрямую практически все форматы исполняемых файлов и операционных систем.
  4. -stdlib=libc++ необходимо указать, поскольку Clang не может сам определить, что FreeBSD на платформе AArch64 использует библиотеку libc++ из LLVM, а не libstdc++ из GCC.
  5. -lz добавлено, чтобы показать, как Clang умеет без проблем находить в sysroot другие библиотеки, в данном случае zlib.

Про Mac OS

К сожалению, macOS больше не поддерживается в LLD, ввиду того, что поддержка формата Mach-O была заброшена несколько лет назад. В связи с этим единственным способом собрать исполняемый файл формата Mach-O является использование линкера ld64 (нативного, или в кросс-системе, если вы сами его соберёте). Хотя утилита ld.bfd из binutils всё ещё его поддерживает.

В качестве заключительного аккорда, мы скопируем собраный бинарь на нашу целевую систему (т.е. на виртуалку, откуда мы извлекли чуть раньше sysroot), и убедимся, что всё работает:

$ rsync zpipe FARM64:"~"
$ ssh FARM64
FreeBSD-ARM64-VM $ chmod +x zpipe
FreeBSD-ARM64-VM $ ldd zpipe
zpipe:
        libz.so.6 => /lib/libz.so.6 (0x4029e000)
        libc++.so.1 => /usr/lib/libc++.so.1 (0x402e4000)
        libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x403da000)
        libm.so.5 => /lib/libm.so.5 (0x40426000)
        libc.so.7 => /lib/libc.so.7 (0x40491000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x408aa000)
FreeBSD-ARM64-VM $ ./zpipe -h
zpipe usage: zpipe [-d] < source > dest

Всё получилось! Теперь мы можем использовать этот кросс-тулчейн для сборки более крупных программ, и ниже я расскажу, как использовать его для сборки реальных проектов.

Опционально: создаём папку с тулчейном LLVM

LLVM предоставляет практически полностью совместимые альтернативы для всех утилит, входящих в binutils (за исключением разве что as), с префиксом llvm- в имени.

Про ассемблер

llvm-mc можно использовать как (очень громоздкий) ассемблер, но он плохо документирован. Подобно gcc, фронтэнд clang также может использоваться как ассемблер, делая as зачастую ненужным.

Наиболее критичной из них является LLD, который полностью подменяет системный линковщик для целевой платформы и может заменить одновременно как ld.bfd из GNU, или gold из GNU/Linux или BSD, так и LINK.EXE от Microsoft при сборке под MSVC. Он поддерживает сборку на (практически) каждой платформе, поддерживаемой LLVM, таким образом устраняя необходимость в нескольких специфических линковщиках.

Оба компилятора, GCC и Clang поддерживают использование ld.lld вместо системного линковщика (которым также может быть и lld, например, на FreeBSD) с помощью ключа -fuse-ld=lld.

На опыте я сталкивался с тем, что драйвер Clang иногда не может найти верный линковщик для некоторых редких платформ, особенно до версии 11.0. По какой-то причине иногда clang открыто игнорирует ключ -fuse-ld=lld и вызывает системный линковщик (в моём случае ld.bfd), который не умеет собирать для AArch64.

Быстрым решением для этой проблемы будет создание папки тулчейна, содержащей символические ссылки, которые переименовывают утилиты LLVM в стандартные программы из binutils:

$  ls -la ~/.llvm/bin/
Permissions Size User  Group Date Modified Name
lrwxrwxrwx    16 marco marco  3 Aug  2020  ar -> /usr/bin/llvm-ar
lrwxrwxrwx    12 marco marco  6 Aug  2020  ld -> /usr/bin/lld
lrwxrwxrwx    21 marco marco  3 Aug  2020  objcopy -> /usr/bin/llvm-objcopy
lrwxrwxrwx    21 marco marco  3 Aug  2020  objdump -> /usr/bin/llvm-objdump
lrwxrwxrwx    20 marco marco  3 Aug  2020  ranlib -> /usr/bin/llvm-ranlib
lrwxrwxrwx    21 marco marco  3 Aug  2020  strings -> /usr/bin/llvm-strings

Также можно использовать ключ -B, который заставляет Clang (или GCC) искать нужные утилиты в этой папке, так что подобная проблема даже не возникает:

$  clang++ -B$HOME/.llvm/bin -stdlib=libc++ --target=aarch64-pc-freebsd --sysroot=$HOME/farm_tree -std=c++17 -o mvd-farm64 mvd.cc
$ file mvd-farm64
mvd-farm64: ELF 64-bit LSB executable, ARM aarch64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 13.0, FreeBSD-style, with debug_info, not stripped

Опционально: создаём обёртки Clang для упрощения кросс-компиляции

Я заметил, что некоторые системы сборки (и под «некоторыми» я имею в виду некоторые кривые Makefile, и иногда Autotools) склонны ломаться, если значения переменных окружения $CC, $CXX или $LD содержат пробелы или несколько параметров. Такое может периодически случаться, если нам необходимо вызвать clang с несколькими аргументами (не говоря уже о тех преступниках, которые хардкодят вызовы gcc в свои билд-скрипты. Впрочем, это уже совсем другая история.)

Понимая, насколько громоздко и сложно каждый раз не забыть выписать все параметры корректно в каждом из случаев, я обычно пишу короткие обёртки для clang и clang++ с целью упростить сборку для конкретной целевой платформы:

$ cat ~/.local/bin/aarch64-pc-freebsd-clang
#!/usr/bin/env sh

exec /usr/bin/clang -B$HOME/.llvm/bin --target=aarch64-pc-freebsd --sysroot=$HOME/farm_tree "$@"
$ cat ~/.local/bin/aarch64-pc-freebsd-clang++
#!/usr/bin/env sh

exec /usr/bin/clang++ -B$HOME/.llvm/bin -stdlib=libc++ --target=aarch64-pc-freebsd --sysroot=$HOME/farm_tree "$@"

Если этот скрипт доступен внутри $PATH, его можно повсеместно использовать как отдельную команду:

$ aarch64-pc-freebsd-clang++ -o tst tst.cc -static
$ file tst
tst: ELF 64-bit LSB executable, ARM aarch64, version 1 (FreeBSD), statically linked, for FreeBSD 13.0 (1300136), FreeBSD-style, with debug_info, not stripped

Autotools, Cmake и Meson, возможно, самые популярные системы сборки для open-source проектов на C и C++ (извини, SCons). Все три поддерживают кросс-компиляцию прямо «из коробки», хотя с некоторыми особенностями.

Autotools

В течение многих лет Autotools славится своей ужасной неуклюжестью и хрупкостью. И хотя эту репутацию он заработал вполне обосновано, он по-прежнему широко используется для большинства крупных проектов GNU. Будучи в строю уже не одно десятилетие, многие его проблемы, если что-то вдруг пошло криво, можно разрешить поиском в сети (хотя если вы пишете свой .ac файл, это не всегда так). По сравнению с более современными системами, для кросс-компиляции ему не нужен специфичный файл тулчейна или какая-то особая конфигурация; всё настраивается исключительно параметрами командной строки.

Скрипт ./configure (созданный утилитой autoconf, или включённый в тарболл с исходниками) обычно поддерживает флаг --host, позволяя пользователю задать триплет целевой системы, на которой предполагается запускать собранные программы и прочие артефакты.

Этот флаг активирует кросс-компиляцию, после чего множество утилит auto-что-то-там начинают выяснять правильный компилятор для целевой платформы, который, как правило, зовётся по именам some-triple-gcc или some-triple-g++.

Попробуем сконфигурировать сборку binutils версии 2.35.1 для платформы aarch-pc-freebsd, используя написанную выше обёртку для вызова Clang:

$ tar xvf binutils-2.35.1.tar.xz
$ mkdir binutils-2.35.1/build # always create a build directory to avoid messing up the source tree
$ cd binutils-2.35.1/build
$ env CC='aarch64-pc-freebsd-clang' CXX='aarch64-pc-freebsd-clang++' AR=llvm-ar ../configure --build=x86_64-pc-linux-gnu --host=aarch64-pc-freebsd --enable-gold=yes
checking build system type... x86_64-pc-linux-gnu
checking host system type... aarch64-pc-freebsd
checking target system type... aarch64-pc-freebsd
checking for a BSD-compatible install... /usr/bin/install -c
checking whether ln works... yes
checking whether ln -s works... yes
checking for a sed that does not truncate output... /usr/bin/sed
checking for gawk... gawk
checking for aarch64-pc-freebsd-gcc... aarch64-pc-freebsd-clang
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... yes
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether aarch64-pc-freebsd-clang accepts -g... yes
checking for aarch64-pc-freebsd-clang option to accept ISO C89... none needed
checking whether we are using the GNU C++ compiler... yes
checking whether aarch64-pc-freebsd-clang++ accepts -g... yes
[...]

Вызов скрипта ./configure выше означает, что я хочу от autotools следующего:

  1. Сконфигурировать сборку на платформе x86_64-pc-linux-gnu (которую я задал с помощью ключа --build);
  2. Собрать программы, которые будут исполняться на платформе aarch64-pc-freebsd, что задаётся ключом --host;
  3. В качестве компиляторов C и C++ использовать обёртки Clang, описанные выше.
  4. В качестве целевой утилиты ar использовать llvm-ar.

Я также задал сборку линковщика Gold, который написан на С++ и может использоваться как неплохой тест на то, как наш импровизированный тулчейн способен компилировать программы на C++.

Если стадия конфигурации не сломается по какой-то причине (вроде не должно), мы можем далее запустить GNU Make и собрать binutils:

$ make -j16 # because I have 16 threads on my system
[ lots of output]
$ mkdir dest
$ make DESTDIR=$PWD/dest install # install into a fake tree

Здесь мы должны получить исполнимые файлы и библиотеки внутри целевой папки, созданной с помощью make install. Быстрая проверка с помощью file подтверждает, что все программы корректно собраны для платформы aarch64-pc-freebsd:

$ file dest/usr/local/bin/ld.gold
dest/usr/local/bin/ld.gold: ELF 64-bit LSB executable, ARM aarch64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 13.0 (1300136), FreeBSD-style, with debug_info, not stripped

CMake

Проще всего настроить CMake для сборки на конкретную платформу можно с помощью файла тулчейна. Он обычно состоит из настроек, которые указывают CMake, как он должен работать с данным тулчейном, определяя такие параметры, как целевая операционная система, архитектура CPU, имя компилятора C++ и т.д.

Для сборки под триплет aarch64-pc-freebsd тулчейн файл может быть таким:

set(CMAKE_SYSTEM_NAME FreeBSD)
set(CMAKE_SYSTEM_PROCESSOR aarch64)

set(CMAKE_SYSROOT $ENV{HOME}/farm_tree)

set(CMAKE_C_COMPILER aarch64-pc-freebsd-clang)
set(CMAKE_CXX_COMPILER aarch64-pc-freebsd-clang++)
set(CMAKE_AR llvm-ar)

# these variables tell CMake to avoid using any binary it finds in 
# the sysroot, while picking headers and libraries exclusively from it 
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)

В этом файле я назначил вышеупомянутую обёртку как кросс-компилятор кода C и C++ для целевой платформы. Также можно использовать напрямую Clang с подходящими параметрами, но это не настолько прямолинейно и потенциально более подвержено ошибкам.

В любом случае, исключительно важно задать правильные значения для переменных CMAKE_SYSROOT и CMAKE_FIND_ROOT_PATH_MODE_*, иначе CMake может ошибочно взять пакеты от текущей платформы, с закономерно неверным результатом.

Далее остаётся лишь указать путь к этому файлу, при конфигурировании сборки, через переменную CMAKE_TOOLCHAIN_FILE, либо через параметр --toolchain. Для примера я соберу пакет {fmt} (это удивительная библиотека C++, которую вам непременно стоит попробовать) для платформы aarch64-pc-freebsd:

$  git clone https://github.com/fmtlib/fmt
Cloning into 'fmt'...
remote: Enumerating objects: 45, done.
remote: Counting objects: 100% (45/45), done.
remote: Compressing objects: 100% (33/33), done.
remote: Total 24446 (delta 17), reused 12 (delta 7), pack-reused 24401
Receiving objects: 100% (24446/24446), 12.08 MiB | 2.00 MiB/s, done.
Resolving deltas: 100% (16551/16551), done.
$ cd fmt
$ cmake -B build -G Ninja -DCMAKE_TOOLCHAIN_FILE=$HOME/toolchain-aarch64-freebsd.cmake -DBUILD_SHARED_LIBS=ON -DFMT_TEST=OFF .
-- CMake version: 3.19.4
-- The CXX compiler identification is Clang 11.0.1
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /home/marco/.local/bin/aarch64-pc-freebsd-clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Version: 7.1.3
-- Build type: Release
-- CXX_STANDARD: 11
-- Performing Test has_std_11_flag
-- Performing Test has_std_11_flag - Success
-- Performing Test has_std_0x_flag
-- Performing Test has_std_0x_flag - Success
-- Performing Test SUPPORTS_USER_DEFINED_LITERALS
-- Performing Test SUPPORTS_USER_DEFINED_LITERALS - Success
-- Performing Test FMT_HAS_VARIANT
-- Performing Test FMT_HAS_VARIANT - Success
-- Required features: cxx_variadic_templates
-- Performing Test HAS_NULLPTR_WARNING
-- Performing Test HAS_NULLPTR_WARNING - Success
-- Looking for strtod_l
-- Looking for strtod_l - not found
-- Configuring done
-- Generating done
-- Build files have been written to: /home/marco/fmt/build

По сравнению с Autotools, командная строка, переданная в cmake очень проста и не требует дополнительных объяснений. После конфигурирования остаётся лишь скомпилировать проект, а затем запустить ninja или make, чтобы установить куда-нибудь получившиеся артефакты.

$ cmake --build build
[4/4] Creating library symlink libfmt.so.7 libfmt.so
$ mkdir dest
$ env DESTDIR=$PWD/dest cmake --build build -- install
[0/1] Install the project...
-- Install configuration: "Release"
-- Installing: /home/marco/fmt/dest/usr/local/lib/libfmt.so.7.1.3
-- Installing: /home/marco/fmt/dest/usr/local/lib/libfmt.so.7
-- Installing: /home/marco/fmt/dest/usr/local/lib/libfmt.so
-- Installing: /home/marco/fmt/dest/usr/local/lib/cmake/fmt/fmt-config.cmake
-- Installing: /home/marco/fmt/dest/usr/local/lib/cmake/fmt/fmt-config-version.cmake
-- Installing: /home/marco/fmt/dest/usr/local/lib/cmake/fmt/fmt-targets.cmake
-- Installing: /home/marco/fmt/dest/usr/local/lib/cmake/fmt/fmt-targets-release.cmake
-- Installing: /home/marco/fmt/dest/usr/local/include/fmt/args.h
-- Installing: /home/marco/fmt/dest/usr/local/include/fmt/chrono.h
-- Installing: /home/marco/fmt/dest/usr/local/include/fmt/color.h
-- Installing: /home/marco/fmt/dest/usr/local/include/fmt/compile.h
-- Installing: /home/marco/fmt/dest/usr/local/include/fmt/core.h
-- Installing: /home/marco/fmt/dest/usr/local/include/fmt/format.h
-- Installing: /home/marco/fmt/dest/usr/local/include/fmt/format-inl.h
-- Installing: /home/marco/fmt/dest/usr/local/include/fmt/locale.h
-- Installing: /home/marco/fmt/dest/usr/local/include/fmt/os.h
-- Installing: /home/marco/fmt/dest/usr/local/include/fmt/ostream.h
-- Installing: /home/marco/fmt/dest/usr/local/include/fmt/posix.h
-- Installing: /home/marco/fmt/dest/usr/local/include/fmt/printf.h
-- Installing: /home/marco/fmt/dest/usr/local/include/fmt/ranges.h
-- Installing: /home/marco/fmt/dest/usr/local/lib/pkgconfig/fmt.pc
$  file dest/usr/local/lib/libfmt.so.7.1.3
dest/usr/local/lib/libfmt.so.7.1.3: ELF 64-bit LSB shared object, ARM aarch64, version 1 (FreeBSD), dynamically linked, for FreeBSD 13.0 (1300136), with debug_info, not stripped

Meson

Подобно CMake, Meson полагается на файлы тулчейнов (так называемые «cross files»), которые определяют, какие программы должны быть использованы для сборки под текущую целевую платформу. Благодаря тому, что они написаны на TOML-подобном языке, они очень просты и понятны:

$ cat meson_aarch64_fbsd_cross.txt
[binaries]
c = '/home/marco/.local/bin/aarch64-pc-freebsd-clang'
cpp = '/home/marco/.local/bin/aarch64-pc-freebsd-clang++'
ld = '/usr/bin/ld.lld'
ar = '/usr/bin/llvm-ar'
objcopy = '/usr/bin/llvm-objcopy'
strip = '/usr/bin/llvm-strip'

[properties]
ld_args = ['--sysroot=/home/marco/farm_tree']

[host_machine]
system = 'freebsd'
cpu_family = 'aarch64'
cpu = 'aarch64'
endian = 'little'

Этот кросс-файл далее нужно указать при вызове meson setup с помощью ключа --cross-file. Остальные настройки абсолютно такие же, как для любой другой сборки с помощью Meson.

Кстати…

Подобным образом можно настроить нативный тулчейн на текущей машине, используя нативный файл с ключом --native-file.

И на этом, пожалуй, всё: подобно CMake весь процесс относительно безболезненен и безошибочен. Для полноты расскажу, как собрать dav1d, декодер VideoLAN AV1, для платформы aarch64-pc-freebsd:

$ git clone https://code.videolan.org/videolan/dav1d
Cloning into 'dav1d'...
warning: redirecting to https://code.videolan.org/videolan/dav1d.git/
remote: Enumerating objects: 164, done.
remote: Counting objects: 100% (164/164), done.
remote: Compressing objects: 100% (91/91), done.
remote: Total 9377 (delta 97), reused 118 (delta 71), pack-reused 9213
Receiving objects: 100% (9377/9377), 3.42 MiB | 54.00 KiB/s, done.
Resolving deltas: 100% (7068/7068), done.
$ meson setup build --cross-file ../meson_aarch64_fbsd_cross.txt --buildtype release
The Meson build system
Version: 0.56.2
Source dir: /home/marco/dav1d
Build dir: /home/marco/dav1d/build
Build type: cross build
Project name: dav1d
Project version: 0.8.1
C compiler for the host machine: /home/marco/.local/bin/aarch64-pc-freebsd-clang (clang 11.0.1 "clang version 11.0.1")
C linker for the host machine: /home/marco/.local/bin/aarch64-pc-freebsd-clang ld.lld 11.0.1
[ output cut ]
$ meson compile -C build
Found runner: ['/usr/bin/ninja']
ninja: Entering directory `build'
[129/129] Linking target tests/seek_stress
$ mkdir dest
$ env DESTDIR=$PWD/dest meson install -C build
ninja: Entering directory `build'
[1/11] Generating vcs_version.h with a custom command
Installing src/libdav1d.so.5.0.1 to /home/marco/dav1d/dest/usr/local/lib
Installing tools/dav1d to /home/marco/dav1d/dest/usr/local/bin
Installing /home/marco/dav1d/include/dav1d/common.h to /home/marco/dav1d/dest/usr/local/include/dav1d
Installing /home/marco/dav1d/include/dav1d/data.h to /home/marco/dav1d/dest/usr/local/include/dav1d
Installing /home/marco/dav1d/include/dav1d/dav1d.h to /home/marco/dav1d/dest/usr/local/include/dav1d
Installing /home/marco/dav1d/include/dav1d/headers.h to /home/marco/dav1d/dest/usr/local/include/dav1d
Installing /home/marco/dav1d/include/dav1d/picture.h to /home/marco/dav1d/dest/usr/local/include/dav1d
Installing /home/marco/dav1d/build/include/dav1d/version.h to /home/marco/dav1d/dest/usr/local/include/dav1d
Installing /home/marco/dav1d/build/meson-private/dav1d.pc to /home/marco/dav1d/dest/usr/local/lib/pkgconfig
$ file dest/usr/local/bin/dav1d
dest/usr/local/bin/dav1d: ELF 64-bit LSB executable, ARM aarch64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 13.0 (1300136), FreeBSD-style, with debug_info, not stripped

Бонус: статическая сборка с musl и Alpine Linux

Статическая сборка программ на C и C++ иногда может спасти вас от множества проблем совместимости библиотек, особенно когда вы не можете контролировать, что именно будет установлено на той целевой платформе, под которую вы планируете сборку. Однако сборка статических бинарников на GNU/Linux довольно непроста, поскольку Glibc всячески препятствует попыткам слинковаться с ним статически.

Почему?

Система разрешения имён (NSS), встроенная в glibc, является одной из главных причин интенсивного использования функций dlopen()/dlsym(). Это связано с использованием сторонних плагинов, вроде mDNS, используемых для разрешения имён.

Musl — это альтернанивный вариант стандартной библиотеки для Linux, которая гораздо терпимее к статической линковке, и нынче включена в большинство крупных дистрибутивов. Этих пакетов зачастую вполне достаточно для статической сборки вашего кода, по крайней мере пока вы планируете оставаться в рамках чистого C.

Однако если вы планируете собирать C++, либо если вам необходимы дополнительные компоненты, ситуация становится гораздо сложнее и интереснее. Любая библиотека, входящая в GNU/Linux (такая, как libstdc++, libz, libffi и другие) обычно собрана только с Glibc, что означает, что любая библиотека, которую вы хотите использовать, должна быть пересобрана для цели Musl. Это так и для libstdc++, что неизбежно означает либо перекомпиляцию GCC, либо сборку копии libc++ от LLVM.

К счастью, существует несколько дистрибутивов, собранных для платформы «Musl-plus-Linux», наиболее известный из которых Alpine Linux. А потому возможно использовать освоенный нами подход: извлекаем sysroot для платформы x86_64-pc-linux-musl, в котором находится полный комплект библиотек и пакетов собранных для Musl, а затем используем его вместе с Clang для сборки 100% статических программ.

Настройка контейнера с Alpine

Хорошей отправной точкой является тарболл с minirootfs от Alpine, который предназначен специально для контейнеров, а потому очень маленький:

$ wget -qO - https://dl-cdn.alpinelinux.org/alpine/v3.13/releases/x86_64/alpine-minirootfs-3.13.1-x86_64.tar.gz | gunzip | sudo tar xfp - -C ~/alpine_tree

Теперь мы можем сделать chroot внутрь образа ~/alpine_tree и настроить его, установив все нужные нам пакеты. Обычно я предпочитаю использовать systemd-nspawn вместо chroot, поскольку это значительно проще и меньше склоняет к ошибкам.

А ещё

system-nspawn может также служить лёгкой альтернативой виртуальным машинам. С ключом --boot он умеет запускать процесс init внутри контейнера. Вот в этом весьма полезном gist можно научиться создавать загрузочный контейнер для дистрибутивов, основанных на OpenRC (например, Alpine)

$ sudo systemd-nspawn -D alpine_tree
Spawning container alpinetree on /home/marco/alpine_tree.
Press ^] three times within 1s to kill container.
alpinetree:~#

Здесь мы можем (по желанию) отредактировать /etc/apk/repositories, чтобы переключиться на бранч edge с самыми свежими пакетами, а затем установить нужные пакеты Alpine, содержащие любые статические библиотеки для сборки кода, который мы хотим собрать:

alpinetree:~# cat /etc/apk/repositories
https://dl-cdn.alpinelinux.org/alpine/edge/main
https://dl-cdn.alpinelinux.org/alpine/edge/community
alpinetree:~# apk update
fetch https://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
v3.13.0-1030-gbabf0a1684 [https://dl-cdn.alpinelinux.org/alpine/edge/main]
v3.13.0-1035-ga3ac7373fd [https://dl-cdn.alpinelinux.org/alpine/edge/community]
OK: 14029 distinct packages available
alpinetree:~# apk upgrade
OK: 6 MiB in 14 packages
alpinetree:~# apk add g++ libc-dev
(1/14) Installing libgcc (10.2.1_pre1-r3)
(2/14) Installing libstdc++ (10.2.1_pre1-r3)
(3/14) Installing binutils (2.35.1-r1)
(4/14) Installing libgomp (10.2.1_pre1-r3)
(5/14) Installing libatomic (10.2.1_pre1-r3)
(6/14) Installing libgphobos (10.2.1_pre1-r3)
(7/14) Installing gmp (6.2.1-r0)
(8/14) Installing isl22 (0.22-r0)
(9/14) Installing mpfr4 (4.1.0-r0)
(10/14) Installing mpc1 (1.2.1-r0)
(11/14) Installing gcc (10.2.1_pre1-r3)
(12/14) Installing musl-dev (1.2.2-r1)
(13/14) Installing libc-dev (0.7.2-r3)
(14/14) Installing g++ (10.2.1_pre1-r3)
Executing busybox-1.33.0-r1.trigger
OK: 188 MiB in 28 packages
alpinetree:~# apk add zlib-dev zlib-static
(1/3) Installing pkgconf (1.7.3-r0)
(2/3) Installing zlib-dev (1.2.11-r3)
(3/3) Installing zlib-static (1.2.11-r3)
Executing busybox-1.33.0-r1.trigger
OK: 189 MiB in 31 packages

В данном примере я установил g++ и libc-dev, чтобы получить статические копии libstdc++, статическую libc.a (Musl), и их соответствующие заголовочные файлы. Я также установил zlib-dev и zlib-static, чтобы получить заголовки zlib и библиотеку zlib.a. Как общее правило — Alpine обычно предоставляет статические версии в пакетах *-static, и заголовочные файлы в пакетах somepackage-dev.

Но не всегда

К сожалению, по неизвестным мне причинам Alpine не предоставляет статические версии некоторых библиотек (например, libfmt). Поэтому внедрять копию зависимостей в проект является вполне обычной практикой для C++, и потому это довольно несложно.

Также не забывайте каждый раз запускать apk upgrade внутри sysroot, чтобы поддерживать локальную копию Alpine в актуальном состоянии.

Сборка статических программ на C++

Когда всё готово, остаётся лишь вызвать clang++с верным --target и --sysroot:

$ clang++ -B$HOME/.llvm/bin --gcc-toolchain=$HOME/alpine_tree/usr --target=x86_64-alpine-linux-musl --sysroot=$HOME/alpine_tree -L$HOME/alpine_tree/lib -std=c++17 -o zpipe zpipe.cc -lz -static
$ file zpipe
zpipe: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, with debug_info, not stripped

Дополнительный ключ --gcc-toolchain в данном случае опционален. Он поможет, если Clang не найдёт внутри sysroot GCC и различные файлы crt*.o. Дополнительный ключ -L для /lib необходим, поскольку в Alpine библиотеки лежат как в /usr/lib, так и в /lib, а последний не рассматривается по умолчанию в clang, который ожидает, что все библиотеки находятся только в $SYSROOT/usr/lib.

Написание обёртки для статической линковки с Musl и Clang

В пакетах Musl обычно есть готовые обёртки musl-gcc и musl-clang, которые вызывают системные компиляторы для сборки и линковки с альтернативным libc. Для удобства я по-быстрому набросал вот такой скрипт на Perl:

#!/usr/bin/env perl

use strict;
use utf8;
use warnings;
use v5.30;

use List::Util 'any';

my $ALPINE_DIR = $ENV{ALPINE_DIR} // "$ENV{HOME}/alpine_tree";
my $TOOLS_DIR = $ENV{TOOLS_DIR} // "$ENV{HOME}/.llvm/bin";

my $CMD_NAME = $0 =~ /\+\+/ ? 'clang++' : 'clang';
my $STATIC = $0 =~ /static/;

sub clang {
    exec $CMD_NAME, @_ or return 0;
}

sub main {
    my $compile = any { /^\s*-c|-S\s*$/ } @ARGV;

    my @args = (
         qq{-B$TOOLS_DIR},
         qq{--gcc-toolchain=$ALPINE_DIR/usr},
         '--target=x86_64-alpine-linux-musl',
         qq{--sysroot=$ALPINE_DIR},
         qq{-L$ALPINE_DIR/lib},
         @ARGV,
    );

    unshift @args, '-static' if $STATIC and not $compile;

    exit 1 unless clang @args;
}

main;

Это более «продвинутый» вариант обёртки, чем тот, что я привёл выше для FreeBSD AArch64. Например, она подразумевает компиляцию кода на С++, если вызвана по имени clang++, или всегда добавляет ключ -static, если вызвана по символической ссылке, содержащей в имени слово static:

$ ls -la $(which musl-clang++)
lrwxrwxrwx    10 marco marco 26 Jan 21:49  /home/marco/.local/bin/musl-clang++ -> musl-clang
$ ls -la $(which musl-clang++-static)
lrwxrwxrwx    10 marco marco 26 Jan 22:03  /home/marco/.local/bin/musl-clang++-static -> musl-clang
$ musl-clang++-static -std=c++17 -o zpipe zpipe.cc -lz # automatically infers C++ and -static
$ file zpipe
zpipe: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, with debug_info, not stripped

Таким образом, теперь возможно заставить Clang линковать только с ключом -static, если установить $CC в musl-clang-static, что может быть полезно для систем сборки, которые не очень хорошо работают со статической линковкой. По опыту могу сказать, что больше всего проблем с этим доставляют Autotools (иногда), а также кривые Makefiles.

Заключение

Кросс-компиляция кода на C и C++ является, и скорее всего навсегда останется непростой задачей, однако она стала значительно проще, покуда LLVM достиг стабильности и стал широко распространён. Параметр -target в Clang избавил меня от траты бесчисленного множества человеко-часов на бесконечную сборку и пересборку GCC и Binutils.

К сожалению, не всё то золото, что блестит, и так бывает довольно часто. До сих пор встречаются костыли, которые собираются только с помощью GCC из-за грязных GNUизмов (да-да, Glibc, это про тебя). Кросс-компиляция под Windows/MSVC также невозможна ввиду сильной запутанности всего тулчейна Visual Studio.

Кроме того, хотя сборка под целевую платформу путём указания в Clang верного триплета стала гораздо проще, чем когда-то была, она всё равно нервно курит, поглядывая на то, насколько тривиальна кросс-компиляция в случае с Rust или Go.

Из новых языков отдельного упоминания заслуживает Zig, поскольку его целью является также облегчение сборки кода на C и C++ под другие платформы.

Команды zig cc и zig c++ потенциально могут стать удивительным швейцарским ножом для кросс-компиляции, благодаря тому, что Zig содержит в себе clang и множество частей из разных проектов, такие как Glibc, Musl, libc++ и MinGW. Благодаря этому любая нужная библиотека собирается, если надо, прямо на лету:

$ zig c++ --target=x86_64-windows-gnu -o str.exe str.cc
$ file str.exe
str.exe: PE32+ executable (console) x86-64, for MS Windows

Пока, я думаю, это выглядит не так совершенно, однако уже работает прямо как магия. Осмелюсь утверждать, что это настоящая киллер-фича Zig, и она достойна внимания даже тех, кому совсем не интересен сам язык.

Источник

  • Introduction

  • Why?

    • Speed
    • Cost
    • Containers + k8s
  • Rejected Strategies

    • Using x86_64-pc-windows-gnu
    • Using wine to run the MSVC toolchain
  • How?

    • Prerequisites
    • 1. Setup toolchain(s)
    • 2. Acquire Rust std lib
    • 3. Acquire CRT and Windows 10 SDK
    • 4. Override cc defaults
    • 5. Profit
  • Bonus: Headless testing

    • 1. Install
    • 2. Specify runner
    • 3. Test
  • Final image definition

  • Common issues

    • CMake
    • MASM
    • Compiler Target Confusion
  • Conclusion

Introduction

Last November I added a new job to our CI to cross compile our project for x86_64-pc-windows-msvc from an x86_64-unknown-linux-gnu host. I had wanted to blog about that at the time but never got around to it, but after making some changes and improvements last month to this, in addition to writing a new utility, I figured now was as good of a time as any to share some knowledge in this area for those who might be interested.

Why?

Before we get started with the How, I want to talk about why one might want to do this in the first place, as natively targeting Windows is a «known quantity» with the least amount of surprise. While there are reasons beyond the following, my primary use case for why I want to do cross compilation to Windows is our Continuous Delivery pipeline for my main project at Embark.

Speed

It’s fairly common knowledge that, generally speaking, Linux is faster than Windows on equivalent hardware. From faster file I/O to better utilization of high core count machines, and faster process and thread creation, many operations done in a typical CI job such as compilation and linking tend to be faster on Linux. And since I am lazy, I’ll let another blog post about cross compiling Firefox from Linux to Windows actually present some numbers in defense of this assertion.

Cost

Though we’re now running a Windows VM in our on-premise data center for our normal Windows CD jobs, we actually used to run it in GCP. It was 1 VM with a modest 32 CPU count, but the licensing costs (Windows Server is licensed by core) alone accounted for >20% of our total costs for this particular GCP project.

Картинка с сайта: jake-shadle.github.io

While this single VM is not a huge deal relative to the total costs of our project, it’s still a budget item that provides no substantive value, and on principle I’d rather have more/better CPUs, RAM, disk, or GPUs, that provide immediate concrete value in our CI, or just for local development.

Containers + k8s

This one is probably the most subjective, so strap in!

While fast CI is a high priority, it really doesn’t matter how fast it is if it gives unreliable results. Since I am the (mostly) sole maintainer, (which yes, we’re trying to fix) for our CD pipeline in a team of almost 40 people, my goal early on was to get it into a reliably working state that I could easily maintain with a minimal amount of my time, since I have other, more fun, things to do.

The primary way I did this was to build buildkite-jobify (we use Buildkite as our CI provider). This is just a small service that spawns Kubernetes (k8s) jobs for each of the CI jobs we run on Linux, based on configuration from the repo itself.

This has a few advantages and disadvantages over a more typical VM approach, which we use for x86_64-pc-windows-msvc (for now?), x86_64-apple-darwin, and aarch64-apple-darwin.

Pros

  • Consistency — Every job run from the same container image has the exact same starting environment.
  • Versioned — The image definitions are part of our monorepo, as well as the k8s job descriptions, so we get atomic updates of the environment CI jobs execute in with the code itself. This also makes rollbacks trivial if needed.
  • Scalability — Scaling a k8s cluster up or down is fairly easily (especially in eg GKE, because $) as long as you have the compute resources. k8s also makes it easy to specify resource requests so that individual jobs can dynamically spin up on the most appropriate node at the time based on the other workloads currently running on the cluster.
  • Movability — Since k8s is just running containers, it’s trivial to move build jobs between different clusters, for example in our case, from GKE to our on-premise cluster.

Cons

  • Clean builds — Clean builds are quite slow compared to incremental builds, however we mitigate this by using cargo-fetcher for faster crate fetching and sccache for compiler output caching.
  • Startup times — Changing the image used for a build job means that every k8s node that runs an image it doesn’t have needs to pull it before running. For example, the pull can take up to almost 2m for our aarch64-linux-android which is by far our largest image at almost 3GiB (the Android NDK/SDK are incredibly bloated). However, this is generally a one time cost per image per node and we don’t update images so often that it is actually a problem in practice.

Rejected Strategies

Before we get into the how I just wanted to show two other strategies that could be used for cross compilation that you might want to consider if your needs are different than ours.

Using x86_64-pc-windows-gnu

To be honest, I rejected this one pretty much immediately simply because the gnu environment is not the «native» msvc environment for Windows. Targeting x86_64-pc-windows-gnu would not be representative for actual builds used by users, and it would be different from the local builds built by developers on Windows, which made it an unappealing option. That being said, generally speaking, Rust crates tend to support x86_64-pc-windows-gnu fairly well, which as we’ll see later is a good thing due to my chosen strategy.

Using wine to run the MSVC toolchain

I briefly considered using wine to run the various components of the MSVC compiler toolchain, as that would be the most accurate way to match the native compilation for x86_64-pc-windows-msvc. However, we already use LLD when linking on Windows since it is vastly faster than the MSVC linker, so why not just replace the rest of the toolchain while we’re at it? 😉 This kind of contradicts the reasons stated in x86_64-pc-windows-gnu since we’d be changing to a completely different compiler with different codegen, but this tradeoff is actually ok with me for a couple of reasons.

The first reason is that the driving force behind clang-cl, lld-link, and the other parts of LLVM replacing the MSVC toolchain, is so that Chrome can be built with LLVM for all of their target platforms. The size of the Chrome project dwarfs the amount of C/C++ code in our project by a huge margin, and (I assume) includes far more…advanced…C++ code than we depend on, so the risk of mis-compilation or other issues compared to cl.exe seems reasonably low.

And secondly, we’re actively trying to get rid of C/C++ dependencies as the Rust ecosystem matures and provides its own versions of C/C++ libraries we use. For example, at the time of this writing, we use roughly 800k lines of C/C++ code, a large portion of which comes from Physx, which we will, hopefully, be able to replace in the future with something like rapier.

How?

Ok, now that I’ve laid out some reasons why you might want to consider cross compilation to Windows from Linux, let’s see how we can actually do it! I’ll be constructing a container image (in Dockerfile format) as we go that can be used to compile a Rust program. If you’re only targeting C/C++ the broad strokes of this strategy will still be relevant, you’ll just have a tougher time of it because…well, C/C++.

The strategy I chose is to use clang, which, like most compilers based off of LLVM (including rustc), is a native cross compiler, to compile any C/C++ code and assembly. Specifically this means using clang-cl and lld-link so that we, generally, don’t need to modify any C/C++ code to take cross compilation into account.

Prerequisites

If you want to follow along at home, you’ll need to be on Linux (though WSL might work?) with something that can build container images, like docker or podman.

1. Setup toolchain(s)

First thing we need are the actual toolchains needed to compile and link a full Rust project.

# We'll just use the official Rust image rather than build our own from scratch
FROM docker.io/library/rust:1.54.0-slim-bullseye

ENV KEYRINGS /usr/local/share/keyrings

RUN set -eux; \
    mkdir -p $KEYRINGS; \
    apt-get update && apt-get install -y gpg curl; \
    # clang/lld/llvm
    curl --fail https://apt.llvm.org/llvm-snapshot.gpg.key | gpg --dearmor > $KEYRINGS/llvm.gpg; \
    echo "deb [signed-by=$KEYRINGS/llvm.gpg] http://apt.llvm.org/bullseye/ llvm-toolchain-bullseye-13 main" > /etc/apt/sources.list.d/llvm.list;

RUN set -eux; \
    # Skipping all of the "recommended" cruft reduces total images size by ~300MiB
    apt-get update && apt-get install --no-install-recommends -y \
        clang-13 \
        # llvm-ar
        llvm-13 \
        lld-13 \
        # We're using this in step 3
        tar; \
    # ensure that clang/clang++ are callable directly
    ln -s clang-13 /usr/bin/clang && ln -s clang /usr/bin/clang++ && ln -s lld-13 /usr/bin/ld.lld; \
    # We also need to setup symlinks ourselves for the MSVC shims because they aren't in the debian packages
    ln -s clang-13 /usr/bin/clang-cl && ln -s llvm-ar-13 /usr/bin/llvm-lib && ln -s lld-link-13 /usr/bin/lld-link; \
    # Verify the symlinks are correct
    clang++ -v; \
    ld.lld -v; \
    # Doesn't have an actual -v/--version flag, but it still exits with 0
    llvm-lib -v; \
    clang-cl -v; \
    lld-link --version; \
    # Use clang instead of gcc when compiling binaries targeting the host (eg proc macros, build files)
    update-alternatives --install /usr/bin/cc cc /usr/bin/clang 100; \
    update-alternatives --install /usr/bin/c++ c++ /usr/bin/clang++ 100; \
    apt-get remove -y --auto-remove; \
    rm -rf /var/lib/apt/lists/*;

2. Acquire Rust std lib

By default, rustup only installs the native host target of x86_64-unknown-linux-gnu, which we still need to compile build scripts and procedural macros, but since we’re cross compiling we need to add the x86_64-pc-windows-msvc target as well to get the Rust std library. We could also build the standard library ourselves, but that would mean requiring nightly and taking time to compile something that we can just download instead.

# Retrieve the std lib for the target
RUN rustup target add x86_64-pc-windows-msvc

3. Acquire CRT and Windows 10 SDK

In all likelihood, you’ll need the MSVCRT and Windows 10 SDK to compile and link most projects that target Windows. This is problematic because the official way to install them is, frankly, atrocious, in addition to not being redistributable (so no one but Microsoft can provide, say, a tarball with the needed files).

But really, our needs are relatively simple compared to a normal developer on Windows, as we just need the headers and libraries from the typical VS installation. We could if we wanted use the Visual Studio Build Tools from a Windows machine, or if we were feeling adventurous try to get it running under wine (warning: I briefly tried this but it requires .NET shenanigans that at the time were broken under wine) and then create our own tarball with the needed files, but that feels too slow and tedious.

So instead, I just took inspiration from other projects and created my own xwin program to download, decompress, and repackage the MSVCRT and Windows SDK into a form appropriate for cross compilation. This has several advantages over using the official installation methods.

  • No cruft — Since this program is tailored specifically to getting only the files needed for compiling and linking we skip a ton of cruft, some of which you can opt out of, but some of which you cannot with the official installers. For example, even if you never target aarch64-pc-windows-msvc, you will still get all of the libraries needed for it.
  • Faster — In addition to not even downloading stuff we don’t need, all download, decompression, and disk writes are done in parallel. On my home machine with ~11.7MiB download speeds and a Ryzen 3900X I can download, decompress, and «install» the MSVCRT and Windows SDK in about 27 seconds.
  • Fixups — While the CRT is generally fine, the Windows SDK headers and libraries are an absolute mess of casing (seriously, what maniac thought it would be a good idea to capitalize the l in .lib!?), making them fairly useless on a case-sensitive file system. Rather than rely on using a case-insensitive file system on Linux, xwin just adds symlinks as needed, so eg. windows.h -> Windows.h, kernel32.lib -> kernel32.Lib etc.

We have two basic options for how we could get the CRT and SDK, either run xwin directly during image building, or run it separately and tarball the files and upload them to something like GCS and just retrieve them as needed in the future. We’ll just use it directly while building the image since that’s easier.

RUN set -eux; \
    xwin_version="0.1.1"; \
    xwin_prefix="xwin-$xwin_version-x86_64-unknown-linux-musl"; \
    # Install xwin to cargo/bin via github release. Note you could also just use `cargo install xwin`.
    curl --fail -L https://github.com/Jake-Shadle/xwin/releases/download/$xwin_version/$xwin_prefix.tar.gz | tar -xzv -C /usr/local/cargo/bin --strip-components=1 $xwin_prefix/xwin; \
    # Splat the CRT and SDK files to /xwin/crt and /xwin/sdk respectively
    xwin --accept-license 1 splat --output /xwin; \
    # Remove unneeded files to reduce image size
    rm -rf .xwin-cache /usr/local/cargo/bin/xwin;

4. Override cc defaults

cc is the Rust ecosystem’s primary (we’ll get to the most common exception later) way to compile C/C++ code for use in Rust crates. By default it will try and use cl.exe and friends when targeting the msvc environment, but since we don’t have that, we need to inform it what we actually want it to use instead. We also need to provide additional compiler options to clang-cl to avoid common problems when compiling code that assumes that targeting x86_64-pc-windows-msvc can only be done with the MSVC toolchain.

We also need to tell lld where to search for libraries. We could place the libs in one of the default lib directories lld will search in, but that would mean changing the layout of the CRT and SDK library directories, so it’s generally easier to just specify them explicitly instead. We use RUSTFLAGS for this, which does mean that if you are specifying things like -Ctarget-feature=+crt-static in .cargo/config.toml you will need to reapply them in the container image either during image build or by overriding the environment at runtime to get everything working.

# Note that we're using the full target triple for each variable instead of the
# simple CC/CXX/AR shorthands to avoid issues when compiling any C/C++ code for
# build dependencies that need to compile and execute in the host environment
ENV CC_x86_64_pc_windows_msvc="clang-cl" \
    CXX_x86_64_pc_windows_msvc="clang-cl" \
    AR_x86_64_pc_windows_msvc="llvm-lib" \
    # Note that we only disable unused-command-line-argument here since clang-cl
    # doesn't implement all of the options supported by cl, but the ones it doesn't
    # are _generally_ not interesting.
    CL_FLAGS="-Wno-unused-command-line-argument -fuse-ld=lld-link /imsvc/xwin/crt/include /imsvc/xwin/sdk/include/ucrt /imsvc/xwin/sdk/include/um /imsvc/xwin/sdk/include/shared" \
    RUSTFLAGS="-Lnative=/xwin/crt/lib/x86_64 -Lnative=/xwin/sdk/lib/um/x86_64 -Lnative=/xwin/sdk/lib/ucrt/x86_64"

# These are separate since docker/podman won't transform environment variables defined in the same ENV block
ENV CFLAGS_x86_64_pc_windows_msvc="$CL_FLAGS" \
    CXXFLAGS_x86_64_pc_windows_msvc="$CL_FLAGS"

As already noted above in the reasons why we went this route, we use lld-link even when compiling on Windows hosts due to its superior speed over link.exe. So for our project we just set it in our .cargo/config.toml so it’s used regardless of host platform.

[target.x86_64-pc-windows-msvc]
linker = "lld-link" # Note the lack of extension, which means it will work on both Windows and unix style platforms

If you don’t already use lld-link when targeting Windows, you’ll need to add an additional environment variable so that cargo knows what linker to use, otherwise it will default to link.exe.

ENV CARGO_TARGET_X86_64_PC_WINDOWS_MSVC_LINKER=lld-link

5. Profit

Building a container image from this Dockerfile spec should allow you to run containers capable of compiling and linking a Rust project targeting Windows, including any C/C++ code that might be used as a dependency….mostly.

cargo build --target x86_64-pc-windows-msvc

Bonus: Headless testing

Of course, though compiling and linking a Rust project on Linux is one thing, our CD pipeline also needs to run tests! I’ve mentioned wine several times so far as a way you could run Windows programs such as the MSVC toolchain under Linux, so naturally, that’s what we’re going to do with our test executables.

1. Install

Debian tends to update packages at a glacial pace, as in the case of wine where the 5.0.3 version packaged in bullseye is about 9 months out of date. In this case, it actually matters, as some crates, for example mio, rely on relatively recent wine releases to implement features or fix bugs. Since mio is a foundational crate in the Rust ecosystem, we’ll be installing wine’s staging version, which is 6.15 at the time of this writing.

RUN set -eux; \
    curl --fail https://dl.winehq.org/wine-builds/winehq.key | gpg --dearmor > $KEYRINGS/winehq.gpg; \
    echo "deb [signed-by=$KEYRINGS/winehq.gpg] https://dl.winehq.org/wine-builds/debian/ bullseye main" > /etc/apt/sources.list.d/winehq.list; \
    # The way the debian package works requires that we add x86 support, even
    # though we are only going be running x86_64 executables. We could also
    # build from source, but that is out of scope.
    dpkg --add-architecture i386; \
    apt-get update && apt-get install --no-install-recommends -y winehq-staging; \
    apt-get remove -y --auto-remove; \
    rm -rf /var/lib/apt/lists/*;

2. Specify runner

By default, cargo will attempt to run test binaries natively, but luckily this behavior is trivial to override by supplying a single environment variable to tell cargo how it should run each test binary. This method is also how you can run tests for wasm32-unknown-unknown locally via a wasm runtime like wasmtime. 🙂

ENV \
    # wine can be quite spammy with log messages and they're generally uninteresting
    WINEDEBUG="-all" \
    # Use wine to run test executables
    CARGO_TARGET_X86_64_PC_WINDOWS_MSVC_RUNNER="wine"

3. Test

Now we can compile, link, and test Windows executables with just a standard cargo invocation.

cargo test --target x86_64-pc-windows-msvc

Final image definition

Putting it all together, here is an image definition that should allow you to cross compile to Windows and run headless tests, without needing a Windows install at any step.

# We'll just use the official Rust image rather than build our own from scratch
FROM docker.io/library/rust:1.54.0-slim-bullseye

ENV KEYRINGS /usr/local/share/keyrings

RUN set -eux; \
    mkdir -p $KEYRINGS; \
    apt-get update && apt-get install -y gpg curl; \
    # clang/lld/llvm
    curl --fail https://apt.llvm.org/llvm-snapshot.gpg.key | gpg --dearmor > $KEYRINGS/llvm.gpg; \
    # wine
    curl --fail https://dl.winehq.org/wine-builds/winehq.key | gpg --dearmor > $KEYRINGS/winehq.gpg; \
    echo "deb [signed-by=$KEYRINGS/llvm.gpg] http://apt.llvm.org/bullseye/ llvm-toolchain-bullseye-13 main" > /etc/apt/sources.list.d/llvm.list; \
    echo "deb [signed-by=$KEYRINGS/winehq.gpg] https://dl.winehq.org/wine-builds/debian/ bullseye main" > /etc/apt/sources.list.d/winehq.list;

RUN set -eux; \
    dpkg --add-architecture i386; \
    # Skipping all of the "recommended" cruft reduces total images size by ~300MiB
    apt-get update && apt-get install --no-install-recommends -y \
        clang-13 \
        # llvm-ar
        llvm-13 \
        lld-13 \
        # get a recent wine so we can run tests
        winehq-staging \
        # Unpack xwin
        tar; \
    # ensure that clang/clang++ are callable directly
    ln -s clang-13 /usr/bin/clang && ln -s clang /usr/bin/clang++ && ln -s lld-13 /usr/bin/ld.lld; \
    # We also need to setup symlinks ourselves for the MSVC shims because they aren't in the debian packages
    ln -s clang-13 /usr/bin/clang-cl && ln -s llvm-ar-13 /usr/bin/llvm-lib && ln -s lld-link-13 /usr/bin/lld-link; \
    # Verify the symlinks are correct
    clang++ -v; \
    ld.lld -v; \
    # Doesn't have an actual -v/--version flag, but it still exits with 0
    llvm-lib -v; \
    clang-cl -v; \
    lld-link --version; \
    # Use clang instead of gcc when compiling binaries targeting the host (eg proc macros, build files)
    update-alternatives --install /usr/bin/cc cc /usr/bin/clang 100; \
    update-alternatives --install /usr/bin/c++ c++ /usr/bin/clang++ 100; \
    apt-get remove -y --auto-remove; \
    rm -rf /var/lib/apt/lists/*;

# Retrieve the std lib for the target
RUN rustup target add x86_64-pc-windows-msvc

RUN set -eux; \
    xwin_version="0.1.1"; \
    xwin_prefix="xwin-$xwin_version-x86_64-unknown-linux-musl"; \
    # Install xwin to cargo/bin via github release. Note you could also just use `cargo install xwin`.
    curl --fail -L https://github.com/Jake-Shadle/xwin/releases/download/$xwin_version/$xwin_prefix.tar.gz | tar -xzv -C /usr/local/cargo/bin --strip-components=1 $xwin_prefix/xwin; \
    # Splat the CRT and SDK files to /xwin/crt and /xwin/sdk respectively
    xwin --accept-license 1 splat --output /xwin; \
    # Remove unneeded files to reduce image size
    rm -rf .xwin-cache /usr/local/cargo/bin/xwin;

# Note that we're using the full target triple for each variable instead of the
# simple CC/CXX/AR shorthands to avoid issues when compiling any C/C++ code for
# build dependencies that need to compile and execute in the host environment
ENV CC_x86_64_pc_windows_msvc="clang-cl" \
    CXX_x86_64_pc_windows_msvc="clang-cl" \
    AR_x86_64_pc_windows_msvc="llvm-lib" \
    # wine can be quite spammy with log messages and they're generally uninteresting
    WINEDEBUG="-all" \
    # Use wine to run test executables
    CARGO_TARGET_X86_64_PC_WINDOWS_MSVC_RUNNER="wine" \
    # Note that we only disable unused-command-line-argument here since clang-cl
    # doesn't implement all of the options supported by cl, but the ones it doesn't
    # are _generally_ not interesting.
    CL_FLAGS="-Wno-unused-command-line-argument -fuse-ld=lld-link /imsvc/xwin/crt/include /imsvc/xwin/sdk/include/ucrt /imsvc/xwin/sdk/include/um /imsvc/xwin/sdk/include/shared" \
    # Let cargo know what linker to invoke if you haven't already specified it
    # in a .cargo/config.toml file
    CARGO_TARGET_X86_64_PC_WINDOWS_MSVC_LINKER="lld-link" \
    RUSTFLAGS="-Lnative=/xwin/crt/lib/x86_64 -Lnative=/xwin/sdk/lib/um/x86_64 -Lnative=/xwin/sdk/lib/ucrt/x86_64"

# These are separate since docker/podman won't transform environment variables defined in the same ENV block
ENV CFLAGS_x86_64_pc_windows_msvc="$CL_FLAGS" \
    CXXFLAGS_x86_64_pc_windows_msvc="$CL_FLAGS"

# Run wineboot just to setup the default WINEPREFIX so we don't do it every
# container run
RUN wine wineboot --init

Here is a gist with the same dockerfile, and an example of how you can build it. I’m using podman here, but docker should also work.

curl --fail -L -o xwin.dockerfile https://gist.githubusercontent.com/Jake-Shadle/542dfa000a37c4d3c216c976e0fbb973/raw/bf6cff2bd4ad776d3def8520adb5a5c657140a9f/xwin.dockerfile
podman -t xwin -f xwin.dockerfile .

Common issues

Unfortunately, everything is not sunshine and unicorns where cross compiling is concerned, but all of them are solvable, at least in principle.

CMake

It’s not exactly a secret that I am not a fan of CMake. Inexplicably (to me at least), CMake has become the default way to configure and build open source C/C++ code. As I basically only use Rust now, this would normally not bother me, however, many Rust crates still wrap C/C++ libraries, and due to the ubiquitous nature of CMake, a significant minority of those crates just directly use cmake (or worse, direct invocation) to let CMake drive the building of the underlying C/C++ code. This is great excusable when it works, however, in my experience, CMake scripts tend to be a house of cards that falls down at the slightest deviation from the «one true path» intended by the author(s) of the CMake scripts and cross compiling to Windows is a big deviation that not only knocks down the cards but also sets them on fire.

The simplest and most effective solution to CMake issues is to replace it with cc. In some cases like Physx or spirv-tools that can be a fair amount of work, but in many cases it’s not much. The benefits of course extend beyond just making cross compilation easier, it also gets rid of the CMake installation dependency, as well as just making it easier for outside contributors to understand how the C/C++ code is built, since they don’t need to actually crawl through some project’s CMake scripts trying to figure out what the hell is going on, they can just look at the build.rs file instead.

MASM

Unfortunately, we don’t need to worry about just Rust, C, and C++ in some Rust projects, there are also a small number of crates here and there which also use assembly. While in the future this will be able to be handled natively in rustc, we have to deal with the present, and unfortunately the present contains multiple assemblers with incompatible syntax. In the Microsoft toolchain, ml64.exe assembles MASM, and while there are ongoing efforts to get LLVM to assemble MASM via llvm-ml, the fact that the last update I can find is from October 2020, and there is no project page for llvm-ml like there are for other llvm tools, tells me I might be wasting my time trying to get it to work for all assembly that we need to compile.

Luckily, there is a fairly easy workaround for this gap until llvm-ml becomes more mature. Even though we aren’t targeting x86_64-pc-windows-gnu for the reasons stated above, the few projects that we use that use assembly generally do have both a MASM version as well as a GAS version so that people who want to can target x86_64-pc-windows-gnu. However, since cross compilation to Windows from a non-Windows platform is fairly rare, you’ll often need to provide PRs to projects to fix up assumptions made about the target and host being the same. And unfortunately, this niche case also comes with a bit of maintenance burden that maintainers of a project might be uncomfortable with taking since they can’t easily provide coverage, which is a totally fair reason to not merge such a PR.

Compiler Target Confusion

This one is the rarest of all, at least anecdotally, as I only encountered this kind of issue in Physx. Basically, the issue boils down to a project assuming that Windows == MSVC toolchain and Clang != Windows, which can result in (typically) preprocessor logic errors.

For example, here we have a clang specific warning being disabled for a single function, except it’s fenced by both using clang as the compiler as well as targeting Linux, which means targeting Windows won’t disable the warning, and if warnings are treated as errors, we’ll get a compile failure.

@see PxCreateFoundation()
*/
#if PX_CLANG
-#if PX_LINUX
-#pragma clang diagnostic push
-#pragma clang diagnostic ignored "-Wreturn-type-c-linkage"
-#endif // PX_LINUX
+    #pragma clang diagnostic push
+    #pragma clang diagnostic ignored "-Wreturn-type-c-linkage"
#endif // PX_CLANG
PX_C_EXPORT PX_FOUNDATION_API physx::PxFoundation& PX_CALL_CONV PxGetFoundation();
#if PX_CLANG
-#if PX_LINUX
-#pragma clang diagnostic pop
-#endif // PX_LINUX
+    #pragma clang diagnostic pop
#endif // PX_CLANG

namespace physx

For the most part these kinds of problems won’t occur since clang-cl very effectively masquerades as cl, including setting predefined macros like _MSC_VER and others so that a vast majority if C/C++ code that targets Windows «just works».

Conclusion

And there you have it, a practical summary on how to cross compile Rust projects for x86_64-pc-windows from a Linux host or container, I hope you’ve found at least some of this information useful!

As for next steps, my team is rapidly improving our renderer built on top of Vulkan and rust-gpu, but our non-software rasterization testing is mostly limited to a few basic tests on our Mac VMs since they are the only ones with GPUs. While I am curious about getting rendering tests working for Windows under wine, I am also quite hesitant. While wine and Proton have been making big steps and support a large amount of Windows games, we are using fairly bleeding edge parts of Vulkan like ray tracing, and running rendering tests on Linux means you’re now running on top of the Linux GPU drivers rather than the Windows ones, making test results fairly suspect on whether they are actually detecting issues that might be present in a native Windows environment. It could still be fun though!

While it might seem like I hate Windows due to the content of this post, that’s very much not the case. I am comfortable in Windows having used it for 20+ years or so both personally and professionally, I just prefer Linux these days, especially for automated infrastructure like CD which this post is geared towards…

..however, the same cannot be said for Apple/Macs, as I do hate them with the fiery passion of a thousand suns. Maintaining «automated» Mac machines is one of the most deeply unpleasant experiences of my career, one I wouldn’t wish on my worst enemy, but since Macs are one of our primary targets (thankfully iOS is off the table due to «reasons»), we do need to build and test it along with our other targets. So maybe cross compiling to Macs will be in a future post. 😅

Источник

09 Feb 2021

Anyone who ever tried to cross-compile a C/C++ program knows how big a PITA the whole process could be. The main reasons for this sorry state of things are generally how byzantine build systems tend to be when configuring for cross-compilation, and how messy it is to set-up your cross toolchain in the first place.

One of the main culprits in my experience has been the GNU toolchain, the decades-old behemoth upon which the POSIXish world has been built for years.
Like many compilers of yore, GCC and its binutils brethren were never designed with the intent to support multiple targets within a single setup, with he only supported approach being installing a full cross build for each triple you wish to target on any given host.

For instance, assuming you wish to build something for FreeBSD on your Linux machine using GCC, you need:

  • A GCC + binutils install for your host triplet (i.e., x86_64-pc-linux-gnu or similar);
  • A GCC + binutils complete install for your target triplet (i.e. x86_64-unknown-freebsd12.2-gcc, as, nm, etc)
  • A sysroot containing the necessary libraries and headers, which you can either build yourself or promptly steal from a running installation of FreeBSD.

This process is sometimes made simpler by Linux distributions or hardware vendors offering a selection of prepackaged compilers, but this will never suffice due to the sheer amount of possible host-target combinations. This sometimes means you have to build the whole toolchain yourself, something that, unless you rock a quite beefy CPU, tends to be a massive waste of time and power.

Clang as a cross compiler

This annoying limitation is one of the reasons why I got interested in LLVM (and thus Clang), which is by-design a full-fledged cross compiler toolchain and is mostly compatible with GNU. A single install can output and compile code for every supported target, as long as a complete sysroot is available at build time.

I found this to be a game-changer, and, while it can’t still compete in convenience with modern language toolchains (such as Go’s gc and GOARCH/GOOS), it’s night and day better than the rigmarole of setting up GNU toolchains. You can now just fetch whatever your favorite package management system has available in its repositories (as long as it’s not extremely old), and avoid messing around with multiple installs of GCC.

Until a few years ago, the whole process wasn’t as smooth as it could be. Due to LLVM not having a full toolchain yet available, you were still supposed to provide a binutils build specific for your target. While this is generally much more tolerable than building the whole compiler (binutils is relatively fast to build), it was still somewhat of a nuisance, and I’m glad that llvm-mc (LLVM’s integrated assembler) and lld (universal linker) are finally stable and as flexible as the rest of LLVM.

With the toolchain now set, the next step becomes to obtain a sysroot in order to provide the needed headers and libraries to compile and link for your target.

Obtaining a sysroot

A super fast way to find a working system directory for a given OS is to rip it straight out of an existing system (a Docker container image will often also do).
For instance, this is how I used tar through ssh as a quick way to extract a working sysroot from a FreeBSD 13-CURRENT AArch64 VM 1:

$ mkdir ~/farm_tree
$ ssh FARM64 'tar cf - /lib /usr/include /usr/lib /usr/local/lib /usr/local/include' | bsdtar xvf - -C $HOME/farm_tree/

Invoking the cross compiler

With everything set, it’s now only a matter of invoking Clang with the right arguments:

$  clang++ --target=aarch64-pc-freebsd --sysroot=$HOME/farm_tree -fuse-ld=lld -stdlib=libc++ -o zpipe zpipe.cc -lz --verbose
clang version 11.0.1
Target: aarch64-pc-freebsd
Thread model: posix
InstalledDir: /usr/bin
 "/usr/bin/clang-11" -cc1 -triple aarch64-pc-freebsd -emit-obj -mrelax-all -disable-free -disable-llvm-verifier -discard-value-names -main-file-name zpipe.cc -mrelocation-model static -mframe-pointer=non-leaf -fno-rounding-math -mconstructor-aliases -munwind-tables -fno-use-init-array -target-cpu generic -target-feature +neon -target-abi aapcs -fallow-half-arguments-and-returns -fno-split-dwarf-inlining -debugger-tuning=gdb -v -resource-dir /usr/lib/clang/11.0.1 -isysroot /home/marco/farm_tree -internal-isystem /home/marco/farm_tree/usr/include/c++/v1 -fdeprecated-macro -fdebug-compilation-dir /home/marco/dummies/cxx -ferror-limit 19 -fno-signed-char -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -faddrsig -o /tmp/zpipe-54f1b1.o -x c++ zpipe.cc
clang -cc1 version 11.0.1 based upon LLVM 11.0.1 default target x86_64-pc-linux-gnu
#include "..." search starts here:
#include <...> search starts here:
 /home/marco/farm_tree/usr/include/c++/v1
 /usr/lib/clang/11.0.1/include
 /home/marco/farm_tree/usr/include
End of search list.
 "/usr/bin/ld.lld" --sysroot=/home/marco/farm_tree --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --enable-new-dtags -o zpipe /home/marco/farm_tree/usr/lib/crt1.o /home/marco/farm_tree/usr/lib/crti.o /home/marco/farm_tree/usr/lib/crtbegin.o -L/home/marco/farm_tree/usr/lib /tmp/zpipe-54f1b1.o -lz -lc++ -lm -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /home/marco/farm_tree/usr/lib/crtend.o /home/marco/farm_tree/usr/lib/crtn.o
$ file zpipe
zpipe: ELF 64-bit LSB executable, ARM aarch64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 13.0 (1300136), FreeBSD-style, with debug_info, not stripped

In the snipped above, I have managed to compile and link a C++ file into an executable for AArch64 FreeBSD, all while using just the clang and lld I had already installed on my GNU/Linux system.

More in detail:

  1. --target switches the LLVM default target (x86_64-pc-linux-gnu) to aarch64-pc-freebsd, thus enabling cross-compilation.
  2. --sysroot forces Clang to assume the specified path as root when searching headers and libraries, instead of the usual paths. Note that sometimes this setting might not be enough, especially if the target uses GCC and Clang somehow fails to detect its install path. This can be easily fixed by specifying --gcc-toolchain, which clarifies where to search for GCC installations.
  3. -fuse-ld=lld tells Clang to use lld instead whatever default the platform uses. As I will explain below, it’s highly unlikely that the system linker understands foreign targets, while LLD can natively support almost every binary format and OS 2.
  4. -stdlib=libc++ is needed here due to Clang failing to detect that FreeBSD on AArch64 uses LLVM’s libc++ instead of GCC’s libstdc++.
  5. -lz is also specified to show how Clang can also resolve other libraries inside the sysroot without issues, in this case, zlib.

The final test is now to copy the binary to our target system (i.e. the VM we ripped the sysroot from before) and check if it works as expected:

$ rsync zpipe FARM64:"~"
$ ssh FARM64
FreeBSD-ARM64-VM $ chmod +x zpipe
FreeBSD-ARM64-VM $ ldd zpipe
zpipe:
        libz.so.6 => /lib/libz.so.6 (0x4029e000)
        libc++.so.1 => /usr/lib/libc++.so.1 (0x402e4000)
        libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x403da000)
        libm.so.5 => /lib/libm.so.5 (0x40426000)
        libc.so.7 => /lib/libc.so.7 (0x40491000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x408aa000)
FreeBSD-ARM64-VM $ ./zpipe -h
zpipe usage: zpipe [-d] < source > dest

Success! It’s now possible to use this cross toolchain to build larger programs, and below I’ll give a quick example to how to use it to build real projects.

Optional: creating an LLVM toolchain directory

LLVM provides a mostly compatible counterpart for almost every tool shipped by binutils (with the notable exception of as 3), prefixed with llvm-.

The most critical of these is LLD, which is a drop in replacement for a plaform’s system linker, capable to replace both GNU ld.bfd and gold on GNU/Linux or BSD, and Microsoft’s LINK.EXE when targeting MSVC. It supports linking on (almost) every platform supported by LLVM, thus removing the nuisance to have multiple specific linkers installed.

Both GCC and Clang support using ld.lld instead of the system linker (which may well be lld, like on FreeBSD) via the command line switch -fuse-ld=lld.

In my experience, I found that Clang’s driver might get confused when picking the right linker on some uncommon platforms, especially before version 11.0.
For some reason, clang sometimes decided to outright ignore the -fuse-ld=lld switch and picked the system linker (ld.bfd in my case), which does not support AArch64.

A fast solution to this is to create a toolchain directory containing symlinks that rename the LLVM utilities to the standard binutils programs:

$  ls -la ~/.llvm/bin/
Permissions Size User  Group Date Modified Name
lrwxrwxrwx    16 marco marco  3 Aug  2020  ar -> /usr/bin/llvm-ar
lrwxrwxrwx    12 marco marco  6 Aug  2020  ld -> /usr/bin/lld
lrwxrwxrwx    21 marco marco  3 Aug  2020  objcopy -> /usr/bin/llvm-objcopy
lrwxrwxrwx    21 marco marco  3 Aug  2020  objdump -> /usr/bin/llvm-objdump
lrwxrwxrwx    20 marco marco  3 Aug  2020  ranlib -> /usr/bin/llvm-ranlib
lrwxrwxrwx    21 marco marco  3 Aug  2020  strings -> /usr/bin/llvm-strings

The -B switch can then be used to force Clang (or GCC) to search the required tools in this directory, stopping the issue from ever occurring:

$  clang++ -B$HOME/.llvm/bin -stdlib=libc++ --target=aarch64-pc-freebsd --sysroot=$HOME/farm_tree -std=c++17 -o mvd-farm64 mvd.cc
$ file mvd-farm64
mvd-farm64: ELF 64-bit LSB executable, ARM aarch64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 13.0, FreeBSD-style, with debug_info, not stripped

Optional: creating Clang wrappers to simplify cross-compilation

I happened to notice that certain build systems (and with “certain” I mean some poorly written Makefiles and sometimes Autotools) have a tendency to misbehave when $CC, $CXX or $LD contain spaces or multiple parameters. This might become a recurrent issue if we need to invoke clang with several arguments. 4

Given also how unwieldy it is to remember to write all of the parameters correctly everywhere, I usually write quick wrappers for clang and clang++ in order to simplify building for a certain target:

$ cat ~/.local/bin/aarch64-pc-freebsd-clang
#!/usr/bin/env sh

exec /usr/bin/clang -B$HOME/.llvm/bin --target=aarch64-pc-freebsd --sysroot=$HOME/farm_tree "$@"
$ cat ~/.local/bin/aarch64-pc-freebsd-clang++
#!/usr/bin/env sh

exec /usr/bin/clang++ -B$HOME/.llvm/bin -stdlib=libc++ --target=aarch64-pc-freebsd --sysroot=$HOME/farm_tree "$@"

If created in a directory inside $PATH, these script can used everywhere as standalone commands:

$ aarch64-pc-freebsd-clang++ -o tst tst.cc -static
$ file tst
tst: ELF 64-bit LSB executable, ARM aarch64, version 1 (FreeBSD), statically linked, for FreeBSD 13.0 (1300136), FreeBSD-style, with debug_info, not stripped

Autotools, CMake, and Meson are arguably the most popular building systems for C and C++ open source projects (sorry, SCons).
All of three support cross-compiling out of the box, albeit with some caveats.

Autotools

Over the years, Autotools has been famous for being horrendously clunky and breaking easily. While this reputation is definitely well earned, it’s still widely used by most large GNU projects. Given it’s been around for decades, it’s quite easy to find support online when something goes awry (sadly, this is not also true when writing .ac files). When compared to its more modern breathren, it doesn’t require any toolchain file or extra configuration when cross compiling, being only driven by command line options.

A ./configure script (either generated by autoconf or shipped by a tarball alongside source code) usually supports the --host flag, allowing the user to specify the triple of the host on which the final artifacts are meant to be run.

This flags activates cross compilation, and causes the “auto-something” array of tools to try to detect the correct compiler for the target, which it generally assumes to be called some-triple-gcc or some-triple-g++.

For instance, let’s try to configure binutils version 2.35.1 for aarch64-pc-freebsd, using the Clang wrapper introduced above:

$ tar xvf binutils-2.35.1.tar.xz
$ mkdir binutils-2.35.1/build # always create a build directory to avoid messing up the source tree
$ cd binutils-2.35.1/build
$ env CC='aarch64-pc-freebsd-clang' CXX='aarch64-pc-freebsd-clang++' AR=llvm-ar ../configure --build=x86_64-pc-linux-gnu --host=aarch64-pc-freebsd --enable-gold=yes
checking build system type... x86_64-pc-linux-gnu
checking host system type... aarch64-pc-freebsd
checking target system type... aarch64-pc-freebsd
checking for a BSD-compatible install... /usr/bin/install -c
checking whether ln works... yes
checking whether ln -s works... yes
checking for a sed that does not truncate output... /usr/bin/sed
checking for gawk... gawk
checking for aarch64-pc-freebsd-gcc... aarch64-pc-freebsd-clang
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... yes
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether aarch64-pc-freebsd-clang accepts -g... yes
checking for aarch64-pc-freebsd-clang option to accept ISO C89... none needed
checking whether we are using the GNU C++ compiler... yes
checking whether aarch64-pc-freebsd-clang++ accepts -g... yes
[...]

The invocation of ./configure above specifies that I want autotools to:

  1. Configure for building on an x86_64-pc-linux-gnu host (which I specified using --build);
  2. Build binaries that will run on aarch64-pc-freebsd, using the --host switch;
  3. Use the Clang wrappers made above as C and C++ compilers;
  4. Use llvm-ar as the target ar.

I also specified to build the Gold linker, which is written in C++ and it’s a good test for well our improvised toolchain handles compiling C++.

If the configuration step doesn’t fail for some reason (it shouldn’t), it’s now time to run GNU Make to build binutils:

$ make -j16 # because I have 16 theads on my system
[ lots of output]
$ mkdir dest
$ make DESTDIR=$PWD/dest install # install into a fake tree

There should now be executable files and libraries inside of the fake tree generated by make install. A quick test using file confirms they have been correctly built for aarch64-pc-freebsd:

$ file dest/usr/local/bin/ld.gold
dest/usr/local/bin/ld.gold: ELF 64-bit LSB executable, ARM aarch64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 13.0 (1300136), FreeBSD-style, with debug_info, not stripped

CMake

The simplest way to set CMake to configure for an arbitrary target is to write a toolchain file. These usually consist of a list of declarations that instructs CMake on how it is supposed to use a given toolchain, specifying parameters like the target operating system, the CPU architecture, the name of the C++ compiler, and such.

One reasonable toolchain file for the aarch64-pc-freebsd triple written as follows:

set(CMAKE_SYSTEM_NAME FreeBSD)
set(CMAKE_SYSTEM_PROCESSOR aarch64)

set(CMAKE_SYSROOT $ENV{HOME}/farm_tree)

set(CMAKE_C_COMPILER aarch64-pc-freebsd-clang)
set(CMAKE_CXX_COMPILER aarch64-pc-freebsd-clang++)
set(CMAKE_AR llvm-ar)

# these variables tell CMake to avoid using any binary it finds in 
# the sysroot, while picking headers and libraries exclusively from it 
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)

In this file, I specified the wrapper created above as the cross compiler for C and C++ for the target. It should be possible to also use plain Clang with the right arguments, but it’s much less straightforward and potentially more error-prone.

In any case, it is very important to indicate the CMAKE_SYSROOT and CMAKE_FIND_ROOT_PATH_MODE_* variables, or otherwise CMake could wrongly pick packages from the host with disastrous results.

It is now only a matter of setting CMAKE_TOOLCHAIN_FILE with the path to the toolchain file when configuring a project. To better illustrate this, I will now also build {fmt} (which is an amazing C++ library you should definitely use) for aarch64-pc-freebsd:

$  git clone https://github.com/fmtlib/fmt
Cloning into 'fmt'...
remote: Enumerating objects: 45, done.
remote: Counting objects: 100% (45/45), done.
remote: Compressing objects: 100% (33/33), done.
remote: Total 24446 (delta 17), reused 12 (delta 7), pack-reused 24401
Receiving objects: 100% (24446/24446), 12.08 MiB | 2.00 MiB/s, done.
Resolving deltas: 100% (16551/16551), done.
$ cd fmt
$ cmake -B build -G Ninja -DCMAKE_TOOLCHAIN_FILE=$HOME/toolchain-aarch64-freebsd.cmake -DBUILD_SHARED_LIBS=ON -DFMT_TEST=OFF .
-- CMake version: 3.19.4
-- The CXX compiler identification is Clang 11.0.1
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /home/marco/.local/bin/aarch64-pc-freebsd-clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Version: 7.1.3
-- Build type: Release
-- CXX_STANDARD: 11
-- Performing Test has_std_11_flag
-- Performing Test has_std_11_flag - Success
-- Performing Test has_std_0x_flag
-- Performing Test has_std_0x_flag - Success
-- Performing Test SUPPORTS_USER_DEFINED_LITERALS
-- Performing Test SUPPORTS_USER_DEFINED_LITERALS - Success
-- Performing Test FMT_HAS_VARIANT
-- Performing Test FMT_HAS_VARIANT - Success
-- Required features: cxx_variadic_templates
-- Performing Test HAS_NULLPTR_WARNING
-- Performing Test HAS_NULLPTR_WARNING - Success
-- Looking for strtod_l
-- Looking for strtod_l - not found
-- Configuring done
-- Generating done
-- Build files have been written to: /home/marco/fmt/build

Compared with Autotools, the command line passed to cmake is very simple and doesn’t need too much explanation. After the configuration step is finished, it’s only a matter to compile the project and get ninja or make to install the resulting artifacts somewhere.

$ cmake --build build
[4/4] Creating library symlink libfmt.so.7 libfmt.so
$ mkdir dest
$ env DESTDIR=$PWD/dest cmake --build build -- install
[0/1] Install the project...
-- Install configuration: "Release"
-- Installing: /home/marco/fmt/dest/usr/local/lib/libfmt.so.7.1.3
-- Installing: /home/marco/fmt/dest/usr/local/lib/libfmt.so.7
-- Installing: /home/marco/fmt/dest/usr/local/lib/libfmt.so
-- Installing: /home/marco/fmt/dest/usr/local/lib/cmake/fmt/fmt-config.cmake
-- Installing: /home/marco/fmt/dest/usr/local/lib/cmake/fmt/fmt-config-version.cmake
-- Installing: /home/marco/fmt/dest/usr/local/lib/cmake/fmt/fmt-targets.cmake
-- Installing: /home/marco/fmt/dest/usr/local/lib/cmake/fmt/fmt-targets-release.cmake
-- Installing: /home/marco/fmt/dest/usr/local/include/fmt/args.h
-- Installing: /home/marco/fmt/dest/usr/local/include/fmt/chrono.h
-- Installing: /home/marco/fmt/dest/usr/local/include/fmt/color.h
-- Installing: /home/marco/fmt/dest/usr/local/include/fmt/compile.h
-- Installing: /home/marco/fmt/dest/usr/local/include/fmt/core.h
-- Installing: /home/marco/fmt/dest/usr/local/include/fmt/format.h
-- Installing: /home/marco/fmt/dest/usr/local/include/fmt/format-inl.h
-- Installing: /home/marco/fmt/dest/usr/local/include/fmt/locale.h
-- Installing: /home/marco/fmt/dest/usr/local/include/fmt/os.h
-- Installing: /home/marco/fmt/dest/usr/local/include/fmt/ostream.h
-- Installing: /home/marco/fmt/dest/usr/local/include/fmt/posix.h
-- Installing: /home/marco/fmt/dest/usr/local/include/fmt/printf.h
-- Installing: /home/marco/fmt/dest/usr/local/include/fmt/ranges.h
-- Installing: /home/marco/fmt/dest/usr/local/lib/pkgconfig/fmt.pc
$  file dest/usr/local/lib/libfmt.so.7.1.3
dest/usr/local/lib/libfmt.so.7.1.3: ELF 64-bit LSB shared object, ARM aarch64, version 1 (FreeBSD), dynamically linked, for FreeBSD 13.0 (1300136), with debug_info, not stripped

Meson

Like CMake, Meson relies on toolchain files (here called “cross files”) to specify which tools should be used when building for a given target. Thanks to being written in a TOML-like language, they are very straightforward:

$ cat meson_aarch64_fbsd_cross.txt
[binaries]
c = '/home/marco/.local/bin/aarch64-pc-freebsd-clang'
cpp = '/home/marco/.local/bin/aarch64-pc-freebsd-clang++'
ld = '/usr/bin/ld.lld'
ar = '/usr/bin/llvm-ar'
objcopy = '/usr/bin/llvm-objcopy'
strip = '/usr/bin/llvm-strip'

[properties]
ld_args = ['--sysroot=/home/marco/farm_tree']

[host_machine]
system = 'freebsd'
cpu_family = 'aarch64'
cpu = 'aarch64'
endian = 'little'

This cross-file can then be specified to meson setup using the --cross-file option 5, with everything else remaining the same as with every other Meson build.

And, well, this is basically it: like with CMake, the whole process is relatively painless and foolproof.
For the sake of completeness, this is how to build dav1d, VideoLAN’s AV1 decoder, for aarch64-pc-freebsd:

$ git clone https://code.videolan.org/videolan/dav1d
Cloning into 'dav1d'...
warning: redirecting to https://code.videolan.org/videolan/dav1d.git/
remote: Enumerating objects: 164, done.
remote: Counting objects: 100% (164/164), done.
remote: Compressing objects: 100% (91/91), done.
remote: Total 9377 (delta 97), reused 118 (delta 71), pack-reused 9213
Receiving objects: 100% (9377/9377), 3.42 MiB | 54.00 KiB/s, done.
Resolving deltas: 100% (7068/7068), done.
$ meson setup build --cross-file ../meson_aarch64_fbsd_cross.txt --buildtype release
The Meson build system
Version: 0.56.2
Source dir: /home/marco/dav1d
Build dir: /home/marco/dav1d/build
Build type: cross build
Project name: dav1d
Project version: 0.8.1
C compiler for the host machine: /home/marco/.local/bin/aarch64-pc-freebsd-clang (clang 11.0.1 "clang version 11.0.1")
C linker for the host machine: /home/marco/.local/bin/aarch64-pc-freebsd-clang ld.lld 11.0.1
[ output cut ]
$ meson compile -C build
Found runner: ['/usr/bin/ninja']
ninja: Entering directory `build'
[129/129] Linking target tests/seek_stress
$ mkdir dest
$ env DESTDIR=$PWD/dest meson install -C build
ninja: Entering directory `build'
[1/11] Generating vcs_version.h with a custom command
Installing src/libdav1d.so.5.0.1 to /home/marco/dav1d/dest/usr/local/lib
Installing tools/dav1d to /home/marco/dav1d/dest/usr/local/bin
Installing /home/marco/dav1d/include/dav1d/common.h to /home/marco/dav1d/dest/usr/local/include/dav1d
Installing /home/marco/dav1d/include/dav1d/data.h to /home/marco/dav1d/dest/usr/local/include/dav1d
Installing /home/marco/dav1d/include/dav1d/dav1d.h to /home/marco/dav1d/dest/usr/local/include/dav1d
Installing /home/marco/dav1d/include/dav1d/headers.h to /home/marco/dav1d/dest/usr/local/include/dav1d
Installing /home/marco/dav1d/include/dav1d/picture.h to /home/marco/dav1d/dest/usr/local/include/dav1d
Installing /home/marco/dav1d/build/include/dav1d/version.h to /home/marco/dav1d/dest/usr/local/include/dav1d
Installing /home/marco/dav1d/build/meson-private/dav1d.pc to /home/marco/dav1d/dest/usr/local/lib/pkgconfig
$ file dest/usr/local/bin/dav1d
dest/usr/local/bin/dav1d: ELF 64-bit LSB executable, ARM aarch64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 13.0 (1300136), FreeBSD-style, with debug_info, not stripped

Bonus: static linking with musl and Alpine Linux

Statically linking a C or C++ program can sometimes save you a lot of library compatibility headaches, especially when you can’t control what’s going to be installed on whatever you plan to target.
Building static binaries is however quite complex on GNU/Linux, due to Glibc actively discouraging people from linking it statically. 6

Musl is a very compatible standard library implementation for Linux that plays much nicer with static linking, and it is now shipped by most major distributions. These packages often suffice in building your code statically, at least as long as you plan to stick with plain C.

The situation gets much more complicated if you plan to use C++, or if you need additional components. Any library shipped by a GNU/Linux system (like libstdc++, libz, libffi and so on) is usually only built for Glibc, meaning that any library you wish to use must be rebuilt to target Musl. This also applies to libstdc++, which inevitably means either recompiling GCC or building a copy of LLVM’s libc++.

Thankfully, there are several distributions out there that target “Musl-plus-Linux”, everyone’s favorite being Alpine Linux. It is thus possible to apply the same strategy we used above to obtain a x86_64-pc-linux-musl sysroot complete of libraries and packages built for Musl, which can then be used by Clang to generate 100% static executables.

Setting up an Alpine container

A good starting point is the minirootfs tarball provided by Alpine, which is meant for containers and tends to be very small:

$ wget -qO - https://dl-cdn.alpinelinux.org/alpine/v3.13/releases/x86_64/alpine-minirootfs-3.13.1-x86_64.tar.gz | gunzip | sudo tar xfp - -C ~/alpine_tree

It is now possible to chroot inside the image in ~/alpine_tree and set it up, installing all the packages you may need.
I prefer in general to use systemd-nspawn in lieu of chroot due to it being vastly better and less error prone. 7

$ $  sudo systemd-nspawn -D alpine_tree
Spawning container alpinetree on /home/marco/alpine_tree.
Press ^] three times within 1s to kill container.
alpinetree:~#

We can now (optionally) switch to the edge branch of Alpine for newer packages by editing /etc/apk/repositories, and then install the required packages containing any static libraries required by the code we want to build:

alpinetree:~# cat /etc/apk/repositories
https://dl-cdn.alpinelinux.org/alpine/edge/main
https://dl-cdn.alpinelinux.org/alpine/edge/community
alpinetree:~# apk update
fetch https://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
v3.13.0-1030-gbabf0a1684 [https://dl-cdn.alpinelinux.org/alpine/edge/main]
v3.13.0-1035-ga3ac7373fd [https://dl-cdn.alpinelinux.org/alpine/edge/community]
OK: 14029 distinct packages available
alpinetree:~# apk upgrade
OK: 6 MiB in 14 packages
alpinetree:~# apk add g++ libc-dev
(1/14) Installing libgcc (10.2.1_pre1-r3)
(2/14) Installing libstdc++ (10.2.1_pre1-r3)
(3/14) Installing binutils (2.35.1-r1)
(4/14) Installing libgomp (10.2.1_pre1-r3)
(5/14) Installing libatomic (10.2.1_pre1-r3)
(6/14) Installing libgphobos (10.2.1_pre1-r3)
(7/14) Installing gmp (6.2.1-r0)
(8/14) Installing isl22 (0.22-r0)
(9/14) Installing mpfr4 (4.1.0-r0)
(10/14) Installing mpc1 (1.2.1-r0)
(11/14) Installing gcc (10.2.1_pre1-r3)
(12/14) Installing musl-dev (1.2.2-r1)
(13/14) Installing libc-dev (0.7.2-r3)
(14/14) Installing g++ (10.2.1_pre1-r3)
Executing busybox-1.33.0-r1.trigger
OK: 188 MiB in 28 packages
alpinetree:~# apk add zlib-dev zlib-static
(1/3) Installing pkgconf (1.7.3-r0)
(2/3) Installing zlib-dev (1.2.11-r3)
(3/3) Installing zlib-static (1.2.11-r3)
Executing busybox-1.33.0-r1.trigger
OK: 189 MiB in 31 packages

In this case I installed g++ and libc-dev in order to get a static copy of libstdc++, a static libc.a (Musl) and their respective headers. I also installed zlib-dev and zlib-static to install zlib’s headers and libz.a, respectively.
As a general rule, Alpine usually ships static versions available inside -static packages, and headers as somepackage-dev. 8

Also, remember every once in a while to run apk upgrade inside the sysroot in order to keep the local Alpine install up to date.

Compiling static C++ programs

With everything now set, it’s only a matter of running clang++ with the right --target and --sysroot:

$ clang++ -B$HOME/.llvm/bin --gcc-toolchain=$HOME/alpine_tree/usr --target=x86_64-alpine-linux-musl --sysroot=$HOME/alpine_tree -L$HOME/alpine_tree/lib -std=c++17 -o zpipe zpipe.cc -lz -static
$ file zpipe
zpipe: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, with debug_info, not stripped

The extra --gcc-toolchain is optional, but may help solving issues where compilation fails due to Clang not detecting where GCC and the various crt*.o files reside in the sysroot.
The extra -L for /lib is required because Alpine splits its libraries between /usr/lib and /lib, and the latter is not automatically picked up by clang, which both usually expect libraries to be located in $SYSROOT/usr/bin.

Writing a wrapper for static linking with Musl and Clang

Musl packages usually come with the upstream-provided shims musl-gcc and musl-clang, which wrap the system compilers in order to build and link with the alternative libc.
In order to provide a similar level of convenience, I quickly whipped up the following Perl script:

#!/usr/bin/env perl

use strict;
use utf8;
use warnings;
use v5.30;

use List::Util 'any';

my $ALPINE_DIR = $ENV{ALPINE_DIR} // "$ENV{HOME}/alpine_tree";
my $TOOLS_DIR = $ENV{TOOLS_DIR} // "$ENV{HOME}/.llvm/bin";

my $CMD_NAME = $0 =~ /\+\+/ ? 'clang++' : 'clang';
my $STATIC = $0 =~ /static/;

sub clang {
	exec $CMD_NAME, @_ or return 0;
}

sub main {
	my $compile = any { /^\s*-c|-S\s*$/ } @ARGV;

	my @args = (
		 qq{-B$TOOLS_DIR},
		 qq{--gcc-toolchain=$ALPINE_DIR/usr},
		 '--target=x86_64-alpine-linux-musl',
		 qq{--sysroot=$ALPINE_DIR},
		 qq{-L$ALPINE_DIR/lib},
		 @ARGV,
	);

	unshift @args, '-static' if $STATIC and not $compile;

	exit 1 unless clang @args;
}

main;

This wrapper is more refined than the FreeBSD AArch64 wrapper above.
For instance, it can infer C++ if invoked as clang++, or always force -static if called from a symlink containing static in its name:

$ ls -la $(which musl-clang++)
lrwxrwxrwx    10 marco marco 26 Jan 21:49  /home/marco/.local/bin/musl-clang++ -> musl-clang
$ ls -la $(which musl-clang++-static)
lrwxrwxrwx    10 marco marco 26 Jan 22:03  /home/marco/.local/bin/musl-clang++-static -> musl-clang
$ musl-clang++-static -std=c++17 -o zpipe zpipe.cc -lz # automatically infers C++ and -static
$ file zpipe
zpipe: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, with debug_info, not stripped

It is thus possible to force Clang to only ever link -static by setting $CC to musl-clang-static, which can be useful with build systems that don’t play nicely with statically linking. From my experience, the worst offenders in this regard are Autotools (sometimes) and poorly written Makefiles.

Conclusions

Cross-compiling C and C++ is and will probably always be an annoying task, but it has got much better since LLVM became production-ready and widely available. Clang’s -target option has saved me countless man-hours that I would have instead wasted building and re-building GCC and Binutils over and over again.

Alas, all that glitters is not gold, as is often the case. There is still code around that only builds with GCC due to nasty GNUisms (I’m looking at you, Glibc). Cross compiling for Windows/MSVC is also bordeline unfeasible due to how messy the whole Visual Studio toolchain is.

Furthermore, while targeting arbitrary triples with Clang is now definitely simpler that it was, it still pales in comparison to how trivial cross compiling with Rust or Go is.

One special mention among these new languages should go to Zig, and its goal to also make C and C++ easy to build for other platforms.

The zig cc and zig c++ commands have the potential to become an amazing swiss-army knife tool for cross compiling, thanks to Zig shipping a copy of clang and large chunks of projects such as Glibc, Musl, libc++ and MinGW. Any required library is then built on-the-fly when required:

$ zig c++ --target=x86_64-windows-gnu -o str.exe str.cc
$ file str.exe
str.exe: PE32+ executable (console) x86-64, for MS Windows

While I think this is not yet perfect, it already feels almost like magic. I dare to say, this might really become a killer selling point for Zig, making it attractive even for those who are not interested in using the language itself.

Источник

clang-windows-cmake

CMake toolchain for cross compiling Windows binaries on Linux using Clang.

Isn’t that what mingw does already?

Yes, but mingw has some disadvantages. One of which is that it uses its own standard library, which must be statically linked or usually packaged as dlls along with the actual product. This significantly increases the size of the package, especially for smaller programs/libraries.

Clang on the other hand, can use the official SDK provided by Microsoft, which means that it uses the same runtime libraries as MSVC which are preinstalled on most modern Windows versions (if not, it can easily be installed manually).

TL;DR:

  • Pros:

    • Can use the same standard library as MSVC.
    • Uses the same compiler that you could use to compile native binaries on your system.
    • Somewhat officially supported by Microsoft(?)

  • Cons:

    • MUST use the Windows SDK provided by Microsoft.
    • Some compatibility issues with older versions of Clang.

  • So why not just use MSVC directly?

    • You’ll need to use Wine, which adds overhead on top of it.

Requirements

  • CMake 3.7 or later.
  • clang/llvm 8.0.0 or later (latest version is recommended)
  • A copy of MSVC and the Windows SDK.
    • You’ll need a Windows machine to download them, or you can use the vsdownload script from the msvc-wine project.
    • You’ll also need to rename all of the .lib files to be lowercased.

Installation

Clone/download this repo, then edit the config.cmake file to set the path and version of MSVC and the Windows SDK. Note that the path should be absolute.

Usage

i686-clang-windows-cmake and x86_64-clang-windows-cmake are wrappers around CMake which sets the toolchain automatically for you. Simply replace cmake in your command invocation with one of them to use it, depending on the architecture you want to compile for (i686 for 32-bit, x86_64 for 64-bit). Alternatively, you can set the toolchain file manually.

You can use the run.sh script provided in the test folder to compile a test project and see if the compiler works.

Note that Windows system libraries (user32, ole32, etc.) must be linked explicitly if you want to use them.

License

This project is in the public domain.

Источник

  1. Can Clang compile for Windows?
  2. Can Clang cross compile?
  3. Can you use Clang on Linux?
  4. How do you compile with Clang?
  5. Is clang better than GCC?
  6. Does GCC use LLVM?
  7. Is Linux compiled with GCC or Clang?
  8. How do you run a Clang format?
  9. How do I run clang in Ubuntu?

Can Clang compile for Windows?

Getting Clang on Windows

On Windows, it’s easy to install the Clang tools. Just grab the “Clang compiler for Windows,” an optional component of the “Desktop development with C++” workload. This will install everything you need to develop with Clang on Windows.

Can Clang cross compile?

Cross compilation issues

On the other hand, Clang/LLVM is natively a cross-compiler, meaning that one set of programs can compile to all targets by setting the -target option. … So you’ll need special options to help Clang understand what target you’re compiling to, where your tools are, etc.

Can you use Clang on Linux?

Clang is also provided in all major BSD or GNU/Linux distributions as part of their respective packaging systems. From Xcode 4.2, Clang is the default compiler for Mac OS X.

How do you compile with Clang?

2.4.

To compile a C++ program on the command line, run the clang++ compiler as follows: $ scl enable llvm-toolset-6.0 ‘clang++ -o output_file source_file …’ This creates a binary file named output_file in the current working directory. If the -o option is omitted, the clang++ compiler creates a file named a.

Is clang better than GCC?

Clang is much faster and uses far less memory than GCC. Clang aims to provide extremely clear and concise diagnostics (error and warning messages), and includes support for expressive diagnostics. GCC’s warnings are sometimes acceptable, but are often confusing and it does not support expressive diagnostics.

Does GCC use LLVM?

LLVM and the GNU Compiler Collection (GCC) are both compilers. … LLVM is a framework to generate object code from any kind of source code. While LLVM and GCC both support a wide variety languages and libraries, they are licensed and developed differently.

Is Linux compiled with GCC or Clang?

GCC supports more language extensions and more assembly language features than Clang and LLVM. GCC is still the only option for compiling the Linux kernel.

How do you run a Clang format?

You can install clang-format and git-clang-format via npm install -g clang-format . To automatically format a file according to Electron C++ code style, run clang-format -i path/to/electron/file.cc . It should work on macOS/Linux/Windows.

How do I run clang in Ubuntu?

You need to specify an input file after typing clang in the terminal in order to tell clang what code to run. This example uses the clang package from the default Ubuntu repositories (clang-3.8) and the following source code for hello. c.

Источник

Понравилась статья? Поделить с друзьями:
0 0 голоса
Рейтинг статьи
Подписаться
Уведомить о
guest

0 комментариев
Старые
Новые Популярные
Межтекстовые Отзывы
Посмотреть все комментарии
  • Индексатор службы microsoft windows search грузит процессор windows 11
  • Как устанавливать apk на windows phone
  • Windows 10 как открыть корзину через проводник
  • Windows 10 корпоративная 22h2 активация
  • Svchost exe windows vista