diff options
| author | Aki <please@ignore.pl> | 2023-03-26 18:07:18 +0200 |
|---|---|---|
| committer | Aki <please@ignore.pl> | 2023-03-26 18:07:18 +0200 |
| commit | ea823149b20e9c3bec66369a5480353e48869279 (patch) | |
| tree | 7a85b34b790d416198e9932492ae1786b6139bb1 | |
| parent | 7d326b8524cfc4b6fb5a34d12143ccc5df20b28e (diff) | |
| download | ignore.pl-ea823149b20e9c3bec66369a5480353e48869279.zip ignore.pl-ea823149b20e9c3bec66369a5480353e48869279.tar.gz ignore.pl-ea823149b20e9c3bec66369a5480353e48869279.tar.bz2 | |
Rewrote fresh shell templating article
| -rw-r--r-- | aliases.map | 1 | ||||
| -rw-r--r-- | graveyard_of_the_drawings.html | 3 | ||||
| -rw-r--r-- | how_to_create_templates_with_shell_cat_and_envsubst-1.png | bin | 1159 -> 0 bytes | |||
| -rw-r--r-- | how_to_create_templates_with_shell_cat_and_envsubst.html | 72 | ||||
| -rw-r--r-- | how_to_generate_files_from_templates_in_shell-1.png | bin | 0 -> 2747 bytes | |||
| -rw-r--r-- | how_to_generate_files_from_templates_in_shell.html | 156 | ||||
| -rw-r--r-- | index.html | 6 |
7 files changed, 162 insertions, 76 deletions
diff --git a/aliases.map b/aliases.map index 2638316..e68eaab 100644 --- a/aliases.map +++ b/aliases.map @@ -7,3 +7,4 @@ /state_of_starshatter_open_january_2022.html /starshatter_open_dev_update_2022_01.html; /self_update_sine.html /self_update_march_2023.html; /how_to_create_generic_interfaces_by_example_of_storing_files_in_arbitrary_memory.html /playing_around_with_simple_interfaces.html; +/how_to_create_templates_with_shell_cat_and_envsubst.html /how_to_generate_files_from_templates_in_shell.html; diff --git a/graveyard_of_the_drawings.html b/graveyard_of_the_drawings.html index 5f7d29b..55bee17 100644 --- a/graveyard_of_the_drawings.html +++ b/graveyard_of_the_drawings.html @@ -13,7 +13,7 @@ <article> <h1>Graveyard of the Drawings</h1> -<p class="subtitle">Last modified on 2021-07-25 19:21+02:00 +<p class="subtitle">Last modified on 2023-03-26 18:05+02:00 <p>Here are the drawings I made for articles that I decided to remove. No context, no nothing. Just images. Despite the style, I still think that it'd be a little bit of waste to just remove them along the texts and reusing them in different articles is just lazy.</p> @@ -28,4 +28,5 @@ different articles is just lazy.</p> <img src="graveyard_of_the_drawings-9.png"> <img src="graveyard_of_the_drawings-10.png"> <img src="graveyard_of_the_drawings-11.png"> +<img src="graveyard_of_the_drawings-12.png"> </article> diff --git a/how_to_create_templates_with_shell_cat_and_envsubst-1.png b/how_to_create_templates_with_shell_cat_and_envsubst-1.png Binary files differdeleted file mode 100644 index d390eac..0000000 --- a/how_to_create_templates_with_shell_cat_and_envsubst-1.png +++ /dev/null diff --git a/how_to_create_templates_with_shell_cat_and_envsubst.html b/how_to_create_templates_with_shell_cat_and_envsubst.html deleted file mode 100644 index 8eb730d..0000000 --- a/how_to_create_templates_with_shell_cat_and_envsubst.html +++ /dev/null @@ -1,72 +0,0 @@ -<!doctype html> -<html lang="en"> -<meta charset="utf-8"> -<meta name="viewport" content="width=device-width, initial-scale=1"> -<meta name="author" content="aki"> -<meta name="tags" content="tutorial, template, templating, cat, shell, envsubst"> -<link rel="icon" type="image/png" href="favicon.png"> -<link rel="stylesheet" type="text/css" href="style.css"> - -<title>How to Create Templates With Shell, cat and envsubst</title> - -<nav><p><a href="https://ignore.pl">ignore.pl</a></p></nav> - -<article> -<h1>How to Create Templates With Shell, cat and envsubst</h1> -<p class="subtitle">Published on 2020-07-14 20:26:00+02:00 -<p>Now something trivial and fun. Creating templates of documents or configurations in shell. There are two reasons why -I considered doing that instead of using some more verbose utilities. First off, availability - I have POSIX compliant -shell pretty much everywhere I go. It's a big part of my usual environment and because of that I'm used to it. That's -the second reason - frequency I use it with. It's good to mention that it's one of more verbose utilities out there. -<p>Let's start with plain shell. I actually use this method to serve the blog posts via fcgiwrap. I use it with a single -file, but it can accept external files. However those files won't have their content expanded in any way. If you keep it -in a one script file then it's possible. Basically, I'm talking about <code>cat</code>:</p> -<pre> -#!/bin/sh -cat /dev/fd/3 $@ /dev/fd/4 3<<BEFORE 4<<AFTER -<!doctype html> -<html lang="en"> -BEFORE -<script src=""></script> -AFTER -</pre> -<img src="how_to_create_templates_with_shell_cat_and_envsubst-1.png" alt="actual cat"> -<p>What you see here is a combination of -<a href="https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07_04">heredoc</a> and plain -redirection. It's done like this to avoid calling <code>cat</code> more than once. The script simply concatenates BEFORE -heredoc with any arguments passed to the script, and finally with AFTER heredoc. The output is similar to what you see -in source of this page. The template is trimmed for the sake of the example. -<p>Now, now. Heredoc can easily expand variables but if we were to <code>cat before.inl $@ after.inl</code> it wouldn't -work. For that I use -<code><a href="https://www.gnu.org/software/gettext/manual/html_node/envsubst-Invocation.html">envsubst</a></code>. -Consider file called <i>template.conf</i>: -<pre> -server { - listen 80; - server_name $DOMAIN$ALIASES; - root /srv/http/$DOMAIN/public; -} -</pre> -<p>It could be wrapped in cat and heredoc, but for me it's not desired. Let's say I want my configuration templates to -be exactly that not executable scripts. There are three different solutions that I've heard: <code>eval</code>, -<code>sed</code> and <code>envsubst</code>. <code>Eval</code> can lead to dangerous situations and minor problems with -whitespace. <code>Sed</code> is what I believe is the most typical and straight-forward solution. As for the last one, -it goes like this:</p> -<pre> -$ export DOMAIN=example.tld -$ export ALIASES=' www.example.tld' -$ envsubst '$DOMAIN$ALIASES' <template.conf -server { - listen 80; - server_name example.tld www.example.tld; - root /srv/http/example.tld/public; -} -</pre> -<p>First, you set the variables (if they are not yet there). Then call the <code>envsubst</code> with a single variable -and redirect the template file into it. This single variable is called <em>SHELL-FORMAT</em>. It restricts the variables -that will be substituted in the input. It's completely optional, but if you have some <code>$server_name</code> you -don't want to substitute, then it's kind of useful. It actually doesn't have any format, you just need to reference the -variables in it: <code>'$DOMAIN$ALIASES'</code>, <code>'$DOMAIN $ALIASES'</code>, and <code>'$DOMAIN,$ALIASES'</code> -all work the same. -</article> -<script src="https://stats.ignore.pl/track.js"></script> diff --git a/how_to_generate_files_from_templates_in_shell-1.png b/how_to_generate_files_from_templates_in_shell-1.png Binary files differnew file mode 100644 index 0000000..2cacecf --- /dev/null +++ b/how_to_generate_files_from_templates_in_shell-1.png diff --git a/how_to_generate_files_from_templates_in_shell.html b/how_to_generate_files_from_templates_in_shell.html new file mode 100644 index 0000000..7e7a2ba --- /dev/null +++ b/how_to_generate_files_from_templates_in_shell.html @@ -0,0 +1,156 @@ +<!doctype html> +<html lang="en"> +<meta charset="utf-8"> +<meta name="viewport" content="width=device-width, initial-scale=1"> +<meta name="author" content="aki"> +<meta name="tags" content="template, configure, cat, shell, sh, envsubst, sed"> +<link rel="icon" type="image/png" href="favicon.png"> +<link rel="stylesheet" href="style.css"> + +<title>How to Generate Files From Templates in Shell</title> + +<nav><p><a href="https://ignore.pl">ignore.pl</a></p></nav> + +<article> +<h1>How to Generate Files From Templates in Shell</h1> +<p class="subtitle">Published on 2023-03-26 18:05:00+02:00 +<p>This is a total rewrite of an old article. I no longer liked it and my methods have changed. +<p>Generating or "configuring" files from a template is a common occurrence. A prime example from what I usually use is +<a href="https://cmake.org/cmake/help/latest/command/configure_file.html">configure_file</a> in CMake. Another example +would be service configuration files sitting in a staging location before getting deployed (e.g., version controlled +configs for e-mail or web server). Today let's focus on these kinds of use and not verbose template engines for e.g., +HTML. +<p>Common notations to mark replacement spots in templates are: <code>$VARIABLE</code>, <code>${VARIABLE}</code>, and +<code>@VARIABLE@</code>. First two are obviously coming directly from shell-like notation of variable substitution. The +latter exists exactly to be different from these for when we want to create a template that may contain <code>$</code> +notation in the output as part of its natural syntax. In other words: when the syntax of generated file uses +<code>$</code>. + +<h2>Using shell itself</h2> +<p>POSIX-compliant shells support a mechanism called +<a href="https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07_04">heredoc</a>. We can +use it in combination with <b>cat</b>(1): + +<pre> +#!/bin/sh +cat <<CONTENT +server { + listen 80; + server_name $USER.$DOMAIN; + root /srv/http/$UESR/public; +} +CONTENT +</pre> + +<p>This case has an obvious problem. This isn't really a template that I promised. Instead, it is a script that +generates the intended output. Someone could also try to argue about useless use of <b>cat</b> here. +<p>Using <b>cat</b> and heredocs gives us a lot of flexibility. We can wrap some content with a common header and footer +if we want to: + +<pre> +#!/bin/sh +cat /dev/fd/3 $@ /dev/fd/3 3<<HEAD 4<<FOOT +<!doctype html> +<html lang="en"> +HEAD +<script src="script.js"></script> +FOOT +</pre> + +<h2>Using envsubst</h2> +<p>If we want a real template instead of an executable script we can use <b>envsubst</b>(1). This tool is extremely +straight-forward in use: put template in standard input and get substituted text in standard output: + +<pre> +server { + listen 80; + server_name $USER.$DOMAIN; + root /srv/http/$USER/public; +} +</pre> + +<p>Then: + +<pre> +$ export DOMAIN=example.tld +$ envsubst <nginx.conf.in +server { + listen 80; + server_name aki.ignore.pl; + root /srv/http/aki/public; +} +</pre> + +<img src="how_to_generate_files_from_templates_in_shell-1.png" alt="shell substitution"> + +<p><b>Envsubst</b> supports <code>${VARIABLE}</code>, too. +<p>Major potential problem with <b>envsubst</b> is that it substitutes everything as it goes. It doesn't matter whether +the variable exists in the environment or not. This is the usual expected behaviour from shell, but it might not be +well suited for handling any output that uses <code>$</code> in a meaningful way. We can partially workaround it using +<i>SHELL-FORMAT</i> argument: + +<pre> +$ envsubst <em>'$USER, $DOMAIN'</em> <nginx.conf.in >nginx.conf +</pre> + +<p>This limits the substitutions to selected variables. The format of this argument is not important. Whatever is a +conformant variable reference will work: <code>'$USER$DOMAIN'</code>, <code>'$USER $DOMAIN'</code>, +<code>'$USER,$DOMAIN'</code>, and the first example are all equivalent. Just remember to not substitute the variables +when calling <b>envsubst</b> by accident and to put it in one argument (hence why single-quotes are used). + +<h2>Using sed</h2> +<p>Finally, we can use <b>sed</b>(1) to gain even more control over what happens to our templates. This comes at the +cost: <b>sed</b> does not have access to environment variables on its own. Usually, we can find it being used like this: + +<pre> +$ sed "s/@VERSION@/$VERSION/g" <version.h.in >version.h +</pre> + +<p>Shell will substitute <code>$VERSION</code> there with the variable and any use of <code>@VERSION@</code> in template +file will be replaced. Note that <b>sed</b> can replace anything it wants - <code>@</code> are used here to make it more +strict and to make template more readable. +<p>If we are feeling like over-engineering, we can generate script for <b>sed</b> and use that instead. The variable +values may come from anywhere you want at that point, let's use shell: + +<pre> +#!/bin/sh +if tag=$(git describe --tags --exact); then + echo s/@VERSION@/$tag/g +else + echo s/@VERSION@/@BRANCH@-@HASH@/g +fi +echo s/@HASH@/$(git rev-parse --short HEAD)/g +echo s/@BRANCH@/$(git symbolic-ref --short HEAD || echo detached)/g +</pre> + +<p>Here rather than using <b>cat</b> I used <code>echo</code>. Depending on the state of repository it is used in, it +may output something similar to: + +<pre> +s/@VERSION@/@BRANCH@-@HASH@/g +s/@HASH@/4242424/g +s/@BRANCH@/nightly/g +</pre> + +<p>We can then feed it into <b>sed</b>: + +<pre> +$ sed -f <em>subst.sed</em> <version.h.in >version.h +</pre> + +<p>Now, if we want to over-engineer it for real, let's put it into a Makefile: + +<pre> +subst.sed: subst.sed.sh + ./$< >$@ + +%.h: %.h.in subst.sed + sed -f subst.sed <$< >$@ +</pre> + +<h2>Other Alternatives</h2> +<p>Otherwise one could potentially use: <b>perl</b>(1), <b>python</b>(1), <b>awk</b>(1), maybe shell's <code>eval</code> +if feeling adventurous (and malicious, I guess). CMakes <code>configure_file</code> is very nice but is limited to +CMake. I'm starting to feel like it could be a nice weekend project to make a utility after a beer or two. +</article> +<script src="https://stats.ignore.pl/track.js"></script> @@ -37,6 +37,9 @@ software and hardware engineering. I post here my projects, experiments, and wha <section id="posts"> <h2>Posts</h2> <ul> +<li> <a href="how_to_generate_files_from_templates_in_shell.html">How to Generate Files From Templates in + Shell</a><br> + <time>2023-03-26</time> <li> <a href="self_update_march_2023.html">Self Update: March 2023</a><br> <time>2023-03-23</time> <li> <a href="starshatter_open_dev_update_2022_09.html">Starshatter: The Open Source Project - Dev Update - @@ -78,9 +81,6 @@ software and hardware engineering. I post here my projects, experiments, and wha <time>2020-07-28</time> <li> <a href="how_to_archive_with_posix_tar_cpio_and_pax.html">How to Archive With POSIX tar, cpio and pax</a><br> <time>2020-07-22</time> -<li> <a href="how_to_create_templates_with_shell_cat_and_envsubst.html">How to Create Templates With Shell, cat and - envsubst</a><br> - <time>2020-07-14</time> <li> <a href="of_privacy_and_traffic_tracking.html">Of Privacy and Traffic Tracking</a><br> <time>2020-07-11</time> <li> <a href="environments_in_lua_5_2_and_beyond.html">Environments in Lua 5.2 and Beyond</a><br> |
