diff options
| author | Aki <please@ignore.pl> | 2022-06-19 17:33:21 +0200 |
|---|---|---|
| committer | Aki <please@ignore.pl> | 2022-06-19 17:33:21 +0200 |
| commit | 58d63ace4c170f17ab79dde1e363b147b09ed712 (patch) | |
| tree | c3575c38775155fca0b6880256e8d9c0859aea97 /using_pacman_to_manage_emscripten_packages.html | |
| parent | 3d297ec53816add6a4eb3b1d76c205c665ff7877 (diff) | |
| download | ignore.pl-58d63ace4c170f17ab79dde1e363b147b09ed712.zip ignore.pl-58d63ace4c170f17ab79dde1e363b147b09ed712.tar.gz ignore.pl-58d63ace4c170f17ab79dde1e363b147b09ed712.tar.bz2 | |
Published using pacman to manage emscripten packages
Diffstat (limited to 'using_pacman_to_manage_emscripten_packages.html')
| -rw-r--r-- | using_pacman_to_manage_emscripten_packages.html | 255 |
1 files changed, 255 insertions, 0 deletions
diff --git a/using_pacman_to_manage_emscripten_packages.html b/using_pacman_to_manage_emscripten_packages.html new file mode 100644 index 0000000..e128f72 --- /dev/null +++ b/using_pacman_to_manage_emscripten_packages.html @@ -0,0 +1,255 @@ +<!doctype html> +<html lang="en"> +<meta charset="utf-8"> +<meta name="viewport" content="width=device-width, initial-scale=1"> +<meta name="author" content="aki"> +<meta name="tags" content="arch linux, pacman, emscripten, packaging, distribution"> +<link rel="icon" type="image/png" href="cylo.png"> +<link rel="stylesheet" href="style.css"> + +<title>Using pacman to Manage Emscripten Packages</title> + +<nav><p><a href="https://ignore.pl">ignore.pl</a></p></nav> + +<article> +<h1>Using pacman to Manage Emscripten Packages</h1> +<p class="subtitle">Published on 2022-06-19 17:35:00+02:00 +<p>C was created for use with Unix. Quite quickly it became one of the most used programming languages of all time. +After some additional years it made its way into Linux kernel and operating system. To this year it is the primary +language that is used to interface with the kernel or to write any sort of utilities. If not directly then through +various bindings. +<p>To allow use of external libraries C has a mechanism for including header files in your own source code. Then during +linking stage compiled implementation of these headers is linked along with your code into final executable (or through +dynamic linker with some additional steps). The configuration of what is visible for including and linking employs use +of several PATH-like variables with some defaults, and sometimes (if you were having a good day and it had to be ruined) +hidden or undocumented behaviour. +<p>Management of the available packages that contain headers and libraries is usually offloaded to the system-wide +package manager. Considering the relation between C and system it's hosted by, it isn't that bad of a choice. Now, it +is not perfect, but with a well maintained upstream and local ecosystems it'll be just right. +<p>Problems may appear when we change or take away one of the parts: operating system, C toolchain, or package manager. +The most prominent examples of when it happens is: Windows, cross-compiling, and porting software between distros. Such +cases, especially the first one, resulted in creation of external package managers e.g., +<a href="https://vcpkg.io/">vcpkg</a>, <a href="https://conan.io/">Conan</a>. In other cases they pushed people toward +build generators such as <a href="https://cmake.org/">CMake</a>.</p> +<img src="using_pacman_to_manage_emscripten_packages-1.png" alt="emscripten logo"> +<p>Recently, I've been playing around with <a href="https://emscripten.org/">Emscripten</a>. I built some things here +and there, and now, you guessed it, I'm trying out different approaches to handling libraries and decided to explore +<a href="https://archlinux.org/pacman/">pacman</a>(1) as means to it. I hope you enjoy this little experiment. +<p>Without going into internals, <b>pacman</b> is a package manager used by <a href="https://archlinux.org/">Arch +Linux</a>, a distribution that describes itself as lightweight, flexible, and simple. It focuses on bleeding-edge +packages. I picked it because I happen to use it on a daily basis. +<p>Packages are distributed in a binary form and come from remote repositories. Package is an archive that contains +files meant for installation and some meta information, all built by +<a href="https://archlinux.org/pacman/makepkg.8.html">makepkg</a>(8). Repository is really just a set of files managed +by a <a href="https://archlinux.org/pacman/repo-add.8.html">repo-add</a>(8). + + +<h3>Building Sample Package</h3> +<p>I started by creating a sample package that provides <a href="https://www.raylib.com/">raylib</a>. To do that, I +wrote a rather simple <code>PKGBUILD</code> file: + +<pre> +pkgname=raylib +pkgver=4.0.0 +pkgrel=1 +arch=(wasm32) +license=(zlib) +makedepends=(cmake emscripten) +source=("${pkgname}.tar.gz::https://github.com/raysan5/raylib/archive/refs/tags/${pkgver}.tar.gz") +sha256sums=("11f6087dc7bedf9efb3f69c0c872f637e421d914e5ecea99bbe7781f173dc38c") +</pre> + +<p>Stop, right now! If you are a seasoned package maintainer or maybe you just cross-compiled enough software, you will +notice that something is not right in here. Yeah, <code>arch</code> is wrong. It's a little bit counter-intuitive, so +take a look at another example +<a href="https://archlinux.org/packages/community/any/aarch64-linux-gnu-glibc/">aarch64-linux-gnu-glibc</a>, GNU C +Library for ARM64 targets: + +<pre> +$ asp checkout aarch64-linux-gnu-glibc +$ cd aarch64-linux-gnu-glibc/trunk +$ grep arch= PKGBUILD +arch=(any) +</pre> + +<p>This is different for a good reason: none of this is going to be used on the host system. Only the compiler and any +binutils will be used, and they are actually targeted for the architecture of build host: <i>x86_64</i> in this case. +<p>Then why am I specifying <i>wasm32</i> for my package? +<p>Emscripten uses cache directories that contain a copy of sysroot. Host system may contain several caches and each +will have own sysroot. I'm not entirely sure what is the reasoning behind it, but that's how it looks like at the moment +of writing. +<p><b>glibc</b> package specifies <i>any</i> architecture, because it is intended to be installed in +<code>/usr/aarch64-linux-gnu</code> and that's where compiler is expecting to see it. I could technically try to make +my package operate in similar manner and install to <code>/usr/lib/emscripten/system</code> that acts as base for caches +and is provided by emscripten package from Arch Linux repositories. I didn't do that because I wanted installed packages +to be immediately available in my cache. To accomplish that, I decided to use <b>pacman</b> similarly to when you +bootstrap a new system installation, and because the package is technically targeted at <i>wasm32</i> I wrote that in +<code>PKGBUILD</code>. +<p>I think the normal way is also worth exploring. Assuming, that I first figure out how to deal with caches, why +emscripten package does not install to usual <code>/usr/wasm32-emscripten</code>, and how to handle propagation of +packages.</p> +<img src="using_pacman_to_manage_emscripten_packages-2.png" alt="pacman, heh"> +<p>Anyway, I went the other way and I had to hack my way through. Let's continue with <code>PKGBUILD</code>: + +<pre> +build() { + cd "${pkgname}-${pkgver}" + emcmake cmake . -B build \ + -DPLATFORM=Web \ + -DBUILD_EXAMPLES=OFF \ + -DCMAKE_INSTALL_PREFIX=/usr + cd build + make +} + +package() { + cd "${pkgname}-${pkgver}/build" + make DESTDIR="${pkgdir}" install + cd .. + install -Dm644 LICENSE "${pkgdir}/usr/share/licenses/${pkgname}/LICENSE" +} +</pre> + +<p>I use CMake wrapper from Emscripten tools. The only part that's worth noting is that by default, CMake with +Emscripten would set <code>CMAKE_INSTALL_PREFIX</code> to the path of currently used cache directory. That's not +feasible for staging packages meant for distribution, so I use plain <code>/usr</code> instead. Thing is Emscripten uses +<code>include</code> and <code>lib</code> directories located directly in the sysroot and not <code>/usr</code>, so I +will need to adjust it somehow at later stage. Not now because Raylib uses GNUInstallDirs, which expands <code>/</code> +prefix to <code>/usr</code>. +<p>Package is ready to be build: + +<pre> +$ makepkg --printsrcinfo >.SRCINFO +$ CFLAGS='' CARCH=wasm32 makepkg +==> Making package: raylib 4.0.0-1 +==> ... +==> Finished making: raylib 4.0.0-1 +$ ls *.pkg.tar.zst +raylib-4.0.0-1-wasm32.pkg.tar.zst +</pre> + +<p>First off, I unset <code>CFLAGS</code> to avoid default options from <code>/etc/makepkg.conf</code> causing problems. +I also need to set <code>CARCH</code> to inform <b>makepkg</b> that I'm cross-compiling to <i>wasm32</i>. + + +<h3>Setting up Repository</h3> +<p>Now that I had the package, I needed to "distribute" it. Repositories used by <b>pacman</b> are dead simple. They can +be served over HTTP, FTP, or even local files. The structure for all methods is the same and relies on file system, +paths, and central database file. The whole setup was: + +<pre> +$ mkdir -p <u>repo_path</u>/wasm32/core +$ cd <u>repo_path</u>/wasm32/core +$ mv <u>package_path</u>/raylib-4.0.0-1-wasm32.pkg.tar.zst . +$ repo-add core.db.tar.gz *.pkg.tar.zst +</pre> + +<p>Yeah, that's it. First create a directory for the repository and move there. Path contains both: architecture and +name of the repository. After that move the built package to the same directory, and finally add it to the database that +has the same name as the repository. Now, it's a matter of making <b>pacman</b> use it. + + +<h3>Installing the Package</h3> +<p>This section may contain wrong uses of tools for the sake of experimentation. If you are faint-hearted or feel the +need of saying "this is not how to do it" or "this is not how you use it" without elaborating or suggesting another +direction, then it's probably better for you to not continue or have a drink first. +<p>Before doing anything I fixed the directory structure of cache to match one that <b>pacman</b> expects: + +<pre> +$ cd <u>cache</u>/sysroot +$ mkdir usr +$ mv include lib bin usr +$ ln -s usr/{include,lib,bin} . +</pre> + +<p>Symlinks should make everyone happy for now. +<p>Next step was to create directories used directly by <b>pacman</b>: + +<pre> +$ mkdir -p etc/pacman.d/{gnupg,hooks} var/{cache/pacman,lib/pacman,log} +</pre> + +<p>And finally first thing that's worth attention - config file located at <code>etc/pacman.conf</code>. The plan was to +use <b>pacman</b> in a bootstrap fashion for the sysroot located in cache directory, so I needed to write that in config +terms: + +<pre> +[options] +RootDir = <u>cache</u>/sysroot/ +CacheDir = <u>cache</u>/sysroot/var/cache/pacman +HookDir = <u>cache</u>/sysroot/etc/pacman.d/hooks +GPGDir = <u>cache</u>/sysroot/etc/pacman.d/gnupg +Architecture = wasm32 +CheckSpace +SigLevel = TrustAll +</pre> + +<p>Some directories were automatically re-rooted and some weren't. I simply experimented with <code>-v</code> option to +see what is used and adjusted config until I ended up with this version. I don't need to mention <code>TrustAll</code>. +Don't do it. +<p>That's not all; repositories also reside in the config file: + +<pre> +[core] +Server = file:///<u>repo_path</u>/$arch/$repo +</pre> + +<p>What's left is to sync database and install package. <b>pacman</b> assumes that it needs to be run as root user, but +because I'm working with a user-owned cache as my root directory I'd prefer to not raise its privileges, especially +considering that misconfiguration could break packages in host system. Let's try it out: + +<pre> +$ fakeroot pacman --config <u>cache</u>/sysroot/etc/pacman.conf -Sy +:: Synchronising package databases... + core 418.0 B 408 KiB/s 00:00 [###################################] 100% +$ fakeroot pacman --config <u>cache</u>/sysroot/etc/pacman.conf -S raylib +resolving dependencies... +looking for conflicting packages... + +Packages (1) raylib-4.0.0-1 + +Total Download Size: 2.04 MiB +Total Installed Size: 4.70 MiB + +:: Proceed with installation? [Y/n] +:: ... +(1/1) installing raylib [###################################] 100% +</pre> + +<p>Looks like the installation process succeeded. Time to try it out. + + +<h3>Trying It Out and Adjusting pkg-config</h3> +<p>Turns out it doesn't work just yet. Some samples would work but not this one. +<p><b>raylib</b> CMake module has a very peculiar way of defining its target. At first it asks <b>pkg-config</b> for +hints and then uses them in a slightly inconsistent way. Long story short, CMake target will have linker options set +based on output from <code>pkg-config --libs --static</code> disregarding any attempts to remove <code>-L</code> +options. +<p>Since I built my package with <code>CMAKE_INSTALL_PREFIX</code> set to <code>/usr</code>, the <code>prefix</code> +variable in installed <code>raylib.pc</code> will be set to <code>/usr</code>. This will result in +<code>-L/usr/lib</code> appearing in public linker options for raylib target, which will break the entire build process. +<p>The problem here is the <code>prefix=/usr</code> in the module definition file. It should point to the actual root +which is located in cache directory. +<p>There are several ways to address it. My favourite was to simply rewrite the prefix as part of install hook that +would be run by <b>pacman</b>. Sadly it failed because hooks are run in chroot. There are ways to fake it, but I didn't +find them worth exploring at that moment. The other way was <code>PKG_CONFIG_SYSROOT_DIR</code>, and that's what I did. +I tried to avoid it due to uncertain situation between <b>pkgconf</b> and <b>pkg-config</b>. +<p>Luckily, it turned out good enough for me to wrap up the whole experiment. I patched <code>Emscripten.cmake</code> +toolchain file and was able to build a sample project that used the installed sample package. +<p>Should I show here something? Nah</p> +<img src="using_pacman_to_manage_emscripten_packages-3.png" alt="tool... chain?"> + + +<h3>Final Notes</h3> +<p>This was a fun experiment. For some reason I really enjoyed that <b>fakeroot</b> use. +<p>Package management or rather dependency management in cross-compilation context sounds like a good next direction to +explore. I found various takes on it. GNU is a little bit more standardized and there are projects like +<a href="https://crosstool-ng.github.io/">crosstool-NG</a> that at the very least ease configuration of toolchains. +I couldn't find many examples of installable binary packages for target with the exception of the standard library. +Instead, it seems that the usual approach is compiling ports by yourself (which is fine) from e.g., incredibly complex +CMake trees (which is fine, but with flames in background). Otherwise, using <b>vcpkg</b> or similar manager. Or doing +something wild. +<p>As for anything else worth noting... I hope I pointed out everything that I wanted in the article itself. If not, +well, it happens. +</article> +<script src="https://stats.ignore.pl/track.js"></script> |