diff options
Diffstat (limited to 'how_to_generate_files_from_templates_in_shell.html')
| -rw-r--r-- | how_to_generate_files_from_templates_in_shell.html | 156 |
1 files changed, 156 insertions, 0 deletions
diff --git a/how_to_generate_files_from_templates_in_shell.html b/how_to_generate_files_from_templates_in_shell.html new file mode 100644 index 0000000..7e7a2ba --- /dev/null +++ b/how_to_generate_files_from_templates_in_shell.html @@ -0,0 +1,156 @@ +<!doctype html> +<html lang="en"> +<meta charset="utf-8"> +<meta name="viewport" content="width=device-width, initial-scale=1"> +<meta name="author" content="aki"> +<meta name="tags" content="template, configure, cat, shell, sh, envsubst, sed"> +<link rel="icon" type="image/png" href="favicon.png"> +<link rel="stylesheet" href="style.css"> + +<title>How to Generate Files From Templates in Shell</title> + +<nav><p><a href="https://ignore.pl">ignore.pl</a></p></nav> + +<article> +<h1>How to Generate Files From Templates in Shell</h1> +<p class="subtitle">Published on 2023-03-26 18:05:00+02:00 +<p>This is a total rewrite of an old article. I no longer liked it and my methods have changed. +<p>Generating or "configuring" files from a template is a common occurrence. A prime example from what I usually use is +<a href="https://cmake.org/cmake/help/latest/command/configure_file.html">configure_file</a> in CMake. Another example +would be service configuration files sitting in a staging location before getting deployed (e.g., version controlled +configs for e-mail or web server). Today let's focus on these kinds of use and not verbose template engines for e.g., +HTML. +<p>Common notations to mark replacement spots in templates are: <code>$VARIABLE</code>, <code>${VARIABLE}</code>, and +<code>@VARIABLE@</code>. First two are obviously coming directly from shell-like notation of variable substitution. The +latter exists exactly to be different from these for when we want to create a template that may contain <code>$</code> +notation in the output as part of its natural syntax. In other words: when the syntax of generated file uses +<code>$</code>. + +<h2>Using shell itself</h2> +<p>POSIX-compliant shells support a mechanism called +<a href="https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07_04">heredoc</a>. We can +use it in combination with <b>cat</b>(1): + +<pre> +#!/bin/sh +cat <<CONTENT +server { + listen 80; + server_name $USER.$DOMAIN; + root /srv/http/$UESR/public; +} +CONTENT +</pre> + +<p>This case has an obvious problem. This isn't really a template that I promised. Instead, it is a script that +generates the intended output. Someone could also try to argue about useless use of <b>cat</b> here. +<p>Using <b>cat</b> and heredocs gives us a lot of flexibility. We can wrap some content with a common header and footer +if we want to: + +<pre> +#!/bin/sh +cat /dev/fd/3 $@ /dev/fd/3 3<<HEAD 4<<FOOT +<!doctype html> +<html lang="en"> +HEAD +<script src="script.js"></script> +FOOT +</pre> + +<h2>Using envsubst</h2> +<p>If we want a real template instead of an executable script we can use <b>envsubst</b>(1). This tool is extremely +straight-forward in use: put template in standard input and get substituted text in standard output: + +<pre> +server { + listen 80; + server_name $USER.$DOMAIN; + root /srv/http/$USER/public; +} +</pre> + +<p>Then: + +<pre> +$ export DOMAIN=example.tld +$ envsubst <nginx.conf.in +server { + listen 80; + server_name aki.ignore.pl; + root /srv/http/aki/public; +} +</pre> + +<img src="how_to_generate_files_from_templates_in_shell-1.png" alt="shell substitution"> + +<p><b>Envsubst</b> supports <code>${VARIABLE}</code>, too. +<p>Major potential problem with <b>envsubst</b> is that it substitutes everything as it goes. It doesn't matter whether +the variable exists in the environment or not. This is the usual expected behaviour from shell, but it might not be +well suited for handling any output that uses <code>$</code> in a meaningful way. We can partially workaround it using +<i>SHELL-FORMAT</i> argument: + +<pre> +$ envsubst <em>'$USER, $DOMAIN'</em> <nginx.conf.in >nginx.conf +</pre> + +<p>This limits the substitutions to selected variables. The format of this argument is not important. Whatever is a +conformant variable reference will work: <code>'$USER$DOMAIN'</code>, <code>'$USER $DOMAIN'</code>, +<code>'$USER,$DOMAIN'</code>, and the first example are all equivalent. Just remember to not substitute the variables +when calling <b>envsubst</b> by accident and to put it in one argument (hence why single-quotes are used). + +<h2>Using sed</h2> +<p>Finally, we can use <b>sed</b>(1) to gain even more control over what happens to our templates. This comes at the +cost: <b>sed</b> does not have access to environment variables on its own. Usually, we can find it being used like this: + +<pre> +$ sed "s/@VERSION@/$VERSION/g" <version.h.in >version.h +</pre> + +<p>Shell will substitute <code>$VERSION</code> there with the variable and any use of <code>@VERSION@</code> in template +file will be replaced. Note that <b>sed</b> can replace anything it wants - <code>@</code> are used here to make it more +strict and to make template more readable. +<p>If we are feeling like over-engineering, we can generate script for <b>sed</b> and use that instead. The variable +values may come from anywhere you want at that point, let's use shell: + +<pre> +#!/bin/sh +if tag=$(git describe --tags --exact); then + echo s/@VERSION@/$tag/g +else + echo s/@VERSION@/@BRANCH@-@HASH@/g +fi +echo s/@HASH@/$(git rev-parse --short HEAD)/g +echo s/@BRANCH@/$(git symbolic-ref --short HEAD || echo detached)/g +</pre> + +<p>Here rather than using <b>cat</b> I used <code>echo</code>. Depending on the state of repository it is used in, it +may output something similar to: + +<pre> +s/@VERSION@/@BRANCH@-@HASH@/g +s/@HASH@/4242424/g +s/@BRANCH@/nightly/g +</pre> + +<p>We can then feed it into <b>sed</b>: + +<pre> +$ sed -f <em>subst.sed</em> <version.h.in >version.h +</pre> + +<p>Now, if we want to over-engineer it for real, let's put it into a Makefile: + +<pre> +subst.sed: subst.sed.sh + ./$< >$@ + +%.h: %.h.in subst.sed + sed -f subst.sed <$< >$@ +</pre> + +<h2>Other Alternatives</h2> +<p>Otherwise one could potentially use: <b>perl</b>(1), <b>python</b>(1), <b>awk</b>(1), maybe shell's <code>eval</code> +if feeling adventurous (and malicious, I guess). CMakes <code>configure_file</code> is very nice but is limited to +CMake. I'm starting to feel like it could be a nice weekend project to make a utility after a beer or two. +</article> +<script src="https://stats.ignore.pl/track.js"></script> |