1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
|
<!doctype html>
<html lang="en">
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="author" content="aki">
<meta name="tags" content="tutorial, structure, organize, organizing, lua, project, files, howto, guide">
<meta name="published-on" content="2021-01-07T15:45:00+01:00">
<meta name="last-modified-on" content="2022-01-26T19:20:00+01:00">
<link rel="icon" type="image/png" href="favicon.png">
<link rel="stylesheet" href="style.css">
<title>How to Organize Your Lua Project</title>
<header>
<nav><a href="https://ignore.pl">ignore.pl</a></nav>
<time>7 January 2021</time>
<h1>How to Organize Your Lua Project</h1>
</header>
<article>
<p>From time to time I hear complaints about how Lua handles modules. Here and there I see and even answer myself
questions regarding <code>require</code> and adjusting the paths in <code>package</code> to allow some desired
behaviour, with the most prominent issue of relative imports that always work.
<p>This is a quite interesting and wide topic. In this article let's focus on the basics - organizing a simple project,
putting it in a development environment, and preparing for the target environment by creating a known and absolute
structure of requiring the modules, hopefully avoiding usually encountered problems entirely.
<p>Before we hop into the explanation of how to organize files in your Lua projects, let's talk about default importing
mechanism in Lua: <a href="https://www.lua.org/manual/5.4/manual.html#pdf-require"><code>require</code></a>.</p>
<img src="how_to_organize_your_lua_project-1.png" alt="lua hierarchy">
<h2>How Does <code>require</code> Handle Paths</h2>
<p>Both <code>package</code> and <code>require</code> are surprisingly interesting tools. At first glance they are
simple. When you look into them, they are still understandable while gaining some complexity that doesn't reach
unnecessary extremes. They are elegant.
<p>They use a mechanism to find desired files called path resolution or usually simply <strong>path</strong>. The main
component of is a sequence of patterns that may become a pathname, e.g.:
<pre>
/usr/lib/lua/?.lua;/usr/lib/lua/?/init.lua
</pre>
<p>What does it tell us? First off, <code>?</code> is going to be replaced by the argument that was provided to the
<code>require</code>. All dots will be replaced by an appropriate path separator so that: <code>a.b.c</code> will
become <code>a/b/c</code> in *nix systems. So, for call like <code>require "a.b.c"</code>, out path will look
like this:
<pre>
/usr/lib/lua/a/b/c.lua;/usr/lib/lua/a/b/c/init.lua
</pre>
<p>Now, each of these paths are tried and the first one that actually exists in the system will be used. If none of them
match an existing file, the import fails. Simple as that.
<p>The path that is used in resolution is set in <code>package.path</code>. You can modify it in Lua, but it is
intrusive and may depend on a single entry point. Generally, if you plan to release your project as a module for people
to use, I encourage you to avoid modifying anything global. And that's global. Anyway, <code>package.path</code> doesn't
appear out of nowhere - it is populated by one of:
<ol>
<li>Environmental variable <code>LUA_PATH_x_x</code>, where <code>x_x</code> is version such as <code>5_4</code>
<li>Environmental variable <code>LUA_PATH</code>
<li>Default as defined in <a href="https://www.lua.org/source/5.4/luaconf.h.html#LUA_PATH_DEFAULT">luaconf.h</a>
</ol>
<p>Interestingly, if two separators <code>;;</code> show up in the environmental variable path, they will be replaced by
the default path. Meaning <code>/path/to/project/?.lua;;</code> works as prepending your custom path to the default one.
<p>Of course, there is way more to it than just this i.a.: requiring modules written in C, searchers or preloads.
However, in our case this knowledge will suffice.
<p>If you are curious how exactly path is loaded be sure to check out
<a href="https://www.lua.org/source/5.4/loadlib.c.html#setpath"><code>setpath</code></a>.
<h2>Production Environment</h2>
<p>To prepare for development, we need to know where we are heading. First step is to consider the execution
environment. Of course, this and packaging are journeys on their own, so let's just look at two common examples: an
application that uses some framework that uses Lua (e.g. a game made in <a href="https://love2d.org/">LÖVE</a>) and a
standalone module for others to use.
<p>In the first case, it's the duty of the framework to configure the path properly and inform you through the
documentation about it. Paths in LÖVE use their own file hierarchy that is managed by <code>love.filesystem</code> and
by default contains both the game's source (directory or the mounted <code>.love</code> archive) and the save directory.
This means that the structure in your source files is directly reflected in the calls to <code>require</code>, so that
<code>require "module.submodule"</code> will always try <code>game/module/submodule.lua</code>, no matter how you run
the game. This case usually doesn't involve any additional environment configuration for the development stage.
<p>In the second case, your project will end up in an already configured environment and will need to fit in. The
installation of the package usually involves copying your files to the directory that is already included in the path,
so that no further configuration is needed for the execution (at least regarding the path). You can assume, that the
successful installation will make your modules available in the way you want them.
<p>This doesn't happen in the development stage, when you rarely install your package, and most certainly you don't
install it each time you want to test it. This means, that you need to adjust the path so that your modules appear in it
as if they were installed in the system. The principle of minimizing the intrusiveness remains, so the best option is to
use the environmental variables to prepare for development. If you run your application or any tool in such environment,
then Lua will have access to your modules no matter where it is run. Additionally, it will be consistent with the
target environment and won't need any additional hacks.
<h2>Development Environment</h2>
<p>All this talk comes down to: set <code>LUA_PATH</code> in your development environment so that it includes your
project files even if they are not installed in system. A simple approach is to source following in each session:
<pre>
export PROJECT=/path/to/project
export LUA_PATH="$PROJECT/?.lua;$PROJECT/?/init.lua;;"
</pre>
<p>Note the double semicolon that will get replaced by default path, so that other modules that are already installed
are also available.
<p>Let's try it out:
<pre>
$ source env.sh
$ find .
./env.sh
./modulea/submodule.lua
./modulea/init.lua
./moduleb.lua
$ cd modulea
$ lua
Lua 5.4.2
> require "moduleb"
table: 0x561e4b72fb20
> require "modulea"
table: 0x561e4b73aa40
> require "modulea.submodule"
table: 0x561e4b743030
</pre>
<p>As you can see, despite being in the subdirectory, you can still use modules with their fully qualified names that
will remain the same once you install the package. Note, that you could <code>require "init"</code> or <code>require
"submodule"</code> in this case, but I strongly recommend against it. Remain specific, follow the rules and pretend that
you use an installed package from an unknown working directory. <strong>Don't</strong> depend on current working
directory as it is not always the same. Using full names that consider the path setup <strong>guarantees
results</strong>.</p>
<img src="how_to_organize_your_lua_project-2.png" alt="a random whale">
<h2>How to Structure Your Lua Project Files</h2>
<p>Finally, this is what we're waiting for. Assume you have a directory that is a parent of all of your project files.
We'll call it a project <strong>root</strong>. Usually, this is also root directory for your version control system, be
it git or anything else, and for other tools such as building systems or even entire IDEs.
<p>Because it is such a central place to the project, I usually just go ahead and prepend it to <code>LUA_PATH</code> in
the very same way as in the section above:
<pre>
export PROJECT=/path/to/project
export LUA_PATH="$PROJECT/?.lua;$PROJECT/?/init.lua;;"
</pre>
<p>Just like previously, any Lua file that will be descendent of the root will be accessible to us through
<code>require</code>. But what is that <i>init.lua</i>?
<p>It's there to create a way to improve hierarchical structure of your project - to allow splitting bigger modules into
smaller parts (or even submodules that could be included on their own), so that the module doesn't grow into a single
millions-lines-long file. <strong>In simpler words</strong>: you can create a directory named after module and put
<i>init.lua</i> file there and it will act just like a sole <i>module.lua</i> in root.
<p>You could also create a directory named after module and <i>module.lua</i> file in root at the same time, but this
way you will have two entries per module in the root instead of just one.
<p>Additionally, you can then put any module-related files into that directory. You can also use <i>init.lua</i> as
a simple wrapper that calls <code>require</code> for each of its submodules and returns a table with them.
<p>Consider a verbose example:
<pre>
$ find .
./conf.lua
./env.sh
./main.lua
./persistence/init.lua
./persistence/tests.lua
./version.lua
./wave/init.lua
./wave/sawtooth.lua
./wave/sine.lua
./wave/square.lua
$ cat wave/init.lua
-- This is a wrapper example.
return {
sawtooth = require "wave.sawtooth",
sine = require "wave.sine",
square = require "wave.square",
}
$ cat persistence/init.lua
-- This is a normal module example.
return {}
$ cat persistance/tests.lua
-- This is a script that tests an example module.
local p = require "persistence"
assert(type(p) == "table")
$ cat main.lua
-- This is an example main of love application.
local persistence = require "persistence"
local wave = require "wave"
$ cat version.lua
-- This is an example module that acts as version string of the application.
return "1.0.0"
</pre>
<p>Now, this is a mash-up of everything we've discussed. Despite it pretending to be LÖVE application it has
<i>env.sh</i>. Why? The reason is simple: the <i>persistence</i> and <i>wave</i> modules are not meant to be distributed
alone, and they won't ever appear in path of any other environment than LÖVE's. But LÖVE is not the only execution
environment in here: <i>persistence/tests.lua</i> is also meant to be executed. Possibly alone through Lua interpreter.
To allow it <i>env.sh</i> is present and used.
<p>Let's have another example of a simple module meant for installation:
<pre>
$ find .
./env.sh
./hello/Class.lua
./hello/init.lua
./hello/tests.lua
./hello/version.lua
./LICENSE
./Makefile
./README
$ cat Makefile
PREFIX?=/usr/local
LIBDIR?=$(PREFIX)/lib
LUADIR?=$(LIBDIR)/lua/5.4
all:
@echo Nothing to be done
install:
cp -r hello $(DESTDIR)$(LUADIR)
uninstall:
rm -fd $(DESTDIR)$(LUADIR)/hello/* $(DESTDIR)$(LUADIR)/hello
</pre>
<p>As you can see <i>Makefile</i> in this example has targets for installation and removal of the package. The structure
again is simple. Root works as part of the resolution path and so our module is placed in it's own directory named after
it.
<p>The last example is a project of a single file module:
<pre>
$ find .
./env.sh
./LICENSE
./object.lua
./README
</pre>
<p>Yes, it's that simple.
<p>Now, having <i>env.sh</i> in every single project might get bothersome, so I usually use a shell function for
managing them, similarly to what Python's venv does or LuaRocks' env. Speaking of,
<a href="https://luarocks.org/">LuaRocks</a> is yet another interesting story to be told.
<h2>Summary</h2>
<ul>
<li>Consider the execution environment when preparing the development environment.
<li>Contain your project. Avoid changing globals if the modules may be used by someone else in an unknown
environment.
<li>Never depend on the current working directory being something specific.
<li>While developing include your project's root in <code>LUA_PATH</code> to influence <code>require</code>'s
behaviour.
<li>Split modules into parts and use <i>init.lua</i> which is available by default to enclose the module into a
single directory.
</ul>
<h2>Alternatives</h2>
<p>This is just one of the ways to handle structuring your Lua project. It's based on simple rules but has broad usage.
One tempting alternative is this little snippet:
<pre>
local parents = (...):match "(.-)[^%.]+$"
require(parents .. "sibling")
</pre>
<p>Another already mentioned alternatives is adjusting <code>package.path</code> directly in Lua. However, I decided to
skip it due to it's intrusiveness.
<p>All in all, Lua is extremely customizable and adjustable. I would be surprised if these three would be the only ways
to organize projects in Lua.
</article>
<script src="https://stats.ignore.pl/track.js"></script>
|
