summaryrefslogtreecommitdiff
path: root/using_pacman_to_manage_emscripten_packages.html
diff options
context:
space:
mode:
authorAki <please@ignore.pl>2022-06-19 17:33:21 +0200
committerAki <please@ignore.pl>2022-06-19 17:33:21 +0200
commit58d63ace4c170f17ab79dde1e363b147b09ed712 (patch)
treec3575c38775155fca0b6880256e8d9c0859aea97 /using_pacman_to_manage_emscripten_packages.html
parent3d297ec53816add6a4eb3b1d76c205c665ff7877 (diff)
downloadignore.pl-58d63ace4c170f17ab79dde1e363b147b09ed712.zip
ignore.pl-58d63ace4c170f17ab79dde1e363b147b09ed712.tar.gz
ignore.pl-58d63ace4c170f17ab79dde1e363b147b09ed712.tar.bz2
Published using pacman to manage emscripten packages
Diffstat (limited to 'using_pacman_to_manage_emscripten_packages.html')
-rw-r--r--using_pacman_to_manage_emscripten_packages.html255
1 files changed, 255 insertions, 0 deletions
diff --git a/using_pacman_to_manage_emscripten_packages.html b/using_pacman_to_manage_emscripten_packages.html
new file mode 100644
index 0000000..e128f72
--- /dev/null
+++ b/using_pacman_to_manage_emscripten_packages.html
@@ -0,0 +1,255 @@
+<!doctype html>
+<html lang="en">
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<meta name="author" content="aki">
+<meta name="tags" content="arch linux, pacman, emscripten, packaging, distribution">
+<link rel="icon" type="image/png" href="cylo.png">
+<link rel="stylesheet" href="style.css">
+
+<title>Using pacman to Manage Emscripten Packages</title>
+
+<nav><p><a href="https://ignore.pl">ignore.pl</a></p></nav>
+
+<article>
+<h1>Using pacman to Manage Emscripten Packages</h1>
+<p class="subtitle">Published on 2022-06-19 17:35:00+02:00
+<p>C was created for use with Unix. Quite quickly it became one of the most used programming languages of all time.
+After some additional years it made its way into Linux kernel and operating system. To this year it is the primary
+language that is used to interface with the kernel or to write any sort of utilities. If not directly then through
+various bindings.
+<p>To allow use of external libraries C has a mechanism for including header files in your own source code. Then during
+linking stage compiled implementation of these headers is linked along with your code into final executable (or through
+dynamic linker with some additional steps). The configuration of what is visible for including and linking employs use
+of several PATH-like variables with some defaults, and sometimes (if you were having a good day and it had to be ruined)
+hidden or undocumented behaviour.
+<p>Management of the available packages that contain headers and libraries is usually offloaded to the system-wide
+package manager. Considering the relation between C and system it's hosted by, it isn't that bad of a choice. Now, it
+is not perfect, but with a well maintained upstream and local ecosystems it'll be just right.
+<p>Problems may appear when we change or take away one of the parts: operating system, C toolchain, or package manager.
+The most prominent examples of when it happens is: Windows, cross-compiling, and porting software between distros. Such
+cases, especially the first one, resulted in creation of external package managers e.g.,
+<a href="https://vcpkg.io/">vcpkg</a>, <a href="https://conan.io/">Conan</a>. In other cases they pushed people toward
+build generators such as <a href="https://cmake.org/">CMake</a>.</p>
+<img src="using_pacman_to_manage_emscripten_packages-1.png" alt="emscripten logo">
+<p>Recently, I've been playing around with <a href="https://emscripten.org/">Emscripten</a>. I built some things here
+and there, and now, you guessed it, I'm trying out different approaches to handling libraries and decided to explore
+<a href="https://archlinux.org/pacman/">pacman</a>(1) as means to it. I hope you enjoy this little experiment.
+<p>Without going into internals, <b>pacman</b> is a package manager used by <a href="https://archlinux.org/">Arch
+Linux</a>, a distribution that describes itself as lightweight, flexible, and simple. It focuses on bleeding-edge
+packages. I picked it because I happen to use it on a daily basis.
+<p>Packages are distributed in a binary form and come from remote repositories. Package is an archive that contains
+files meant for installation and some meta information, all built by
+<a href="https://archlinux.org/pacman/makepkg.8.html">makepkg</a>(8). Repository is really just a set of files managed
+by a <a href="https://archlinux.org/pacman/repo-add.8.html">repo-add</a>(8).
+
+
+<h3>Building Sample Package</h3>
+<p>I started by creating a sample package that provides <a href="https://www.raylib.com/">raylib</a>. To do that, I
+wrote a rather simple <code>PKGBUILD</code> file:
+
+<pre>
+pkgname=raylib
+pkgver=4.0.0
+pkgrel=1
+arch=(wasm32)
+license=(zlib)
+makedepends=(cmake emscripten)
+source=("${pkgname}.tar.gz::https://github.com/raysan5/raylib/archive/refs/tags/${pkgver}.tar.gz")
+sha256sums=("11f6087dc7bedf9efb3f69c0c872f637e421d914e5ecea99bbe7781f173dc38c")
+</pre>
+
+<p>Stop, right now! If you are a seasoned package maintainer or maybe you just cross-compiled enough software, you will
+notice that something is not right in here. Yeah, <code>arch</code> is wrong. It's a little bit counter-intuitive, so
+take a look at another example
+<a href="https://archlinux.org/packages/community/any/aarch64-linux-gnu-glibc/">aarch64-linux-gnu-glibc</a>, GNU C
+Library for ARM64 targets:
+
+<pre>
+$ asp checkout aarch64-linux-gnu-glibc
+$ cd aarch64-linux-gnu-glibc/trunk
+$ grep arch= PKGBUILD
+arch=(any)
+</pre>
+
+<p>This is different for a good reason: none of this is going to be used on the host system. Only the compiler and any
+binutils will be used, and they are actually targeted for the architecture of build host: <i>x86_64</i> in this case.
+<p>Then why am I specifying <i>wasm32</i> for my package?
+<p>Emscripten uses cache directories that contain a copy of sysroot. Host system may contain several caches and each
+will have own sysroot. I'm not entirely sure what is the reasoning behind it, but that's how it looks like at the moment
+of writing.
+<p><b>glibc</b> package specifies <i>any</i> architecture, because it is intended to be installed in
+<code>/usr/aarch64-linux-gnu</code> and that's where compiler is expecting to see it. I could technically try to make
+my package operate in similar manner and install to <code>/usr/lib/emscripten/system</code> that acts as base for caches
+and is provided by emscripten package from Arch Linux repositories. I didn't do that because I wanted installed packages
+to be immediately available in my cache. To accomplish that, I decided to use <b>pacman</b> similarly to when you
+bootstrap a new system installation, and because the package is technically targeted at <i>wasm32</i> I wrote that in
+<code>PKGBUILD</code>.
+<p>I think the normal way is also worth exploring. Assuming, that I first figure out how to deal with caches, why
+emscripten package does not install to usual <code>/usr/wasm32-emscripten</code>, and how to handle propagation of
+packages.</p>
+<img src="using_pacman_to_manage_emscripten_packages-2.png" alt="pacman, heh">
+<p>Anyway, I went the other way and I had to hack my way through. Let's continue with <code>PKGBUILD</code>:
+
+<pre>
+build() {
+ cd "${pkgname}-${pkgver}"
+ emcmake cmake . -B build \
+ -DPLATFORM=Web \
+ -DBUILD_EXAMPLES=OFF \
+ -DCMAKE_INSTALL_PREFIX=/usr
+ cd build
+ make
+}
+
+package() {
+ cd "${pkgname}-${pkgver}/build"
+ make DESTDIR="${pkgdir}" install
+ cd ..
+ install -Dm644 LICENSE "${pkgdir}/usr/share/licenses/${pkgname}/LICENSE"
+}
+</pre>
+
+<p>I use CMake wrapper from Emscripten tools. The only part that's worth noting is that by default, CMake with
+Emscripten would set <code>CMAKE_INSTALL_PREFIX</code> to the path of currently used cache directory. That's not
+feasible for staging packages meant for distribution, so I use plain <code>/usr</code> instead. Thing is Emscripten uses
+<code>include</code> and <code>lib</code> directories located directly in the sysroot and not <code>/usr</code>, so I
+will need to adjust it somehow at later stage. Not now because Raylib uses GNUInstallDirs, which expands <code>/</code>
+prefix to <code>/usr</code>.
+<p>Package is ready to be build:
+
+<pre>
+$ makepkg --printsrcinfo &gt;.SRCINFO
+$ CFLAGS='' CARCH=wasm32 makepkg
+==> Making package: raylib 4.0.0-1
+==> ...
+==> Finished making: raylib 4.0.0-1
+$ ls *.pkg.tar.zst
+raylib-4.0.0-1-wasm32.pkg.tar.zst
+</pre>
+
+<p>First off, I unset <code>CFLAGS</code> to avoid default options from <code>/etc/makepkg.conf</code> causing problems.
+I also need to set <code>CARCH</code> to inform <b>makepkg</b> that I'm cross-compiling to <i>wasm32</i>.
+
+
+<h3>Setting up Repository</h3>
+<p>Now that I had the package, I needed to "distribute" it. Repositories used by <b>pacman</b> are dead simple. They can
+be served over HTTP, FTP, or even local files. The structure for all methods is the same and relies on file system,
+paths, and central database file. The whole setup was:
+
+<pre>
+$ mkdir -p <u>repo_path</u>/wasm32/core
+$ cd <u>repo_path</u>/wasm32/core
+$ mv <u>package_path</u>/raylib-4.0.0-1-wasm32.pkg.tar.zst .
+$ repo-add core.db.tar.gz *.pkg.tar.zst
+</pre>
+
+<p>Yeah, that's it. First create a directory for the repository and move there. Path contains both: architecture and
+name of the repository. After that move the built package to the same directory, and finally add it to the database that
+has the same name as the repository. Now, it's a matter of making <b>pacman</b> use it.
+
+
+<h3>Installing the Package</h3>
+<p>This section may contain wrong uses of tools for the sake of experimentation. If you are faint-hearted or feel the
+need of saying "this is not how to do it" or "this is not how you use it" without elaborating or suggesting another
+direction, then it's probably better for you to not continue or have a drink first.
+<p>Before doing anything I fixed the directory structure of cache to match one that <b>pacman</b> expects:
+
+<pre>
+$ cd <u>cache</u>/sysroot
+$ mkdir usr
+$ mv include lib bin usr
+$ ln -s usr/{include,lib,bin} .
+</pre>
+
+<p>Symlinks should make everyone happy for now.
+<p>Next step was to create directories used directly by <b>pacman</b>:
+
+<pre>
+$ mkdir -p etc/pacman.d/{gnupg,hooks} var/{cache/pacman,lib/pacman,log}
+</pre>
+
+<p>And finally first thing that's worth attention - config file located at <code>etc/pacman.conf</code>. The plan was to
+use <b>pacman</b> in a bootstrap fashion for the sysroot located in cache directory, so I needed to write that in config
+terms:
+
+<pre>
+[options]
+RootDir = <u>cache</u>/sysroot/
+CacheDir = <u>cache</u>/sysroot/var/cache/pacman
+HookDir = <u>cache</u>/sysroot/etc/pacman.d/hooks
+GPGDir = <u>cache</u>/sysroot/etc/pacman.d/gnupg
+Architecture = wasm32
+CheckSpace
+SigLevel = TrustAll
+</pre>
+
+<p>Some directories were automatically re-rooted and some weren't. I simply experimented with <code>-v</code> option to
+see what is used and adjusted config until I ended up with this version. I don't need to mention <code>TrustAll</code>.
+Don't do it.
+<p>That's not all; repositories also reside in the config file:
+
+<pre>
+[core]
+Server = file:///<u>repo_path</u>/$arch/$repo
+</pre>
+
+<p>What's left is to sync database and install package. <b>pacman</b> assumes that it needs to be run as root user, but
+because I'm working with a user-owned cache as my root directory I'd prefer to not raise its privileges, especially
+considering that misconfiguration could break packages in host system. Let's try it out:
+
+<pre>
+$ fakeroot pacman --config <u>cache</u>/sysroot/etc/pacman.conf -Sy
+:: Synchronising package databases...
+ core 418.0 B 408 KiB/s 00:00 [###################################] 100%
+$ fakeroot pacman --config <u>cache</u>/sysroot/etc/pacman.conf -S raylib
+resolving dependencies...
+looking for conflicting packages...
+
+Packages (1) raylib-4.0.0-1
+
+Total Download Size: 2.04 MiB
+Total Installed Size: 4.70 MiB
+
+:: Proceed with installation? [Y/n]
+:: ...
+(1/1) installing raylib [###################################] 100%
+</pre>
+
+<p>Looks like the installation process succeeded. Time to try it out.
+
+
+<h3>Trying It Out and Adjusting pkg-config</h3>
+<p>Turns out it doesn't work just yet. Some samples would work but not this one.
+<p><b>raylib</b> CMake module has a very peculiar way of defining its target. At first it asks <b>pkg-config</b> for
+hints and then uses them in a slightly inconsistent way. Long story short, CMake target will have linker options set
+based on output from <code>pkg-config --libs --static</code> disregarding any attempts to remove <code>-L</code>
+options.
+<p>Since I built my package with <code>CMAKE_INSTALL_PREFIX</code> set to <code>/usr</code>, the <code>prefix</code>
+variable in installed <code>raylib.pc</code> will be set to <code>/usr</code>. This will result in
+<code>-L/usr/lib</code> appearing in public linker options for raylib target, which will break the entire build process.
+<p>The problem here is the <code>prefix=/usr</code> in the module definition file. It should point to the actual root
+which is located in cache directory.
+<p>There are several ways to address it. My favourite was to simply rewrite the prefix as part of install hook that
+would be run by <b>pacman</b>. Sadly it failed because hooks are run in chroot. There are ways to fake it, but I didn't
+find them worth exploring at that moment. The other way was <code>PKG_CONFIG_SYSROOT_DIR</code>, and that's what I did.
+I tried to avoid it due to uncertain situation between <b>pkgconf</b> and <b>pkg-config</b>.
+<p>Luckily, it turned out good enough for me to wrap up the whole experiment. I patched <code>Emscripten.cmake</code>
+toolchain file and was able to build a sample project that used the installed sample package.
+<p>Should I show here something? Nah</p>
+<img src="using_pacman_to_manage_emscripten_packages-3.png" alt="tool... chain?">
+
+
+<h3>Final Notes</h3>
+<p>This was a fun experiment. For some reason I really enjoyed that <b>fakeroot</b> use.
+<p>Package management or rather dependency management in cross-compilation context sounds like a good next direction to
+explore. I found various takes on it. GNU is a little bit more standardized and there are projects like
+<a href="https://crosstool-ng.github.io/">crosstool-NG</a> that at the very least ease configuration of toolchains.
+I couldn't find many examples of installable binary packages for target with the exception of the standard library.
+Instead, it seems that the usual approach is compiling ports by yourself (which is fine) from e.g., incredibly complex
+CMake trees (which is fine, but with flames in background). Otherwise, using <b>vcpkg</b> or similar manager. Or doing
+something wild.
+<p>As for anything else worth noting... I hope I pointed out everything that I wanted in the article itself. If not,
+well, it happens.
+</article>
+<script src="https://stats.ignore.pl/track.js"></script>