Organizing Your Lua Project

Published on 2021-01-07 15:45:00+01:00

From time to time I hear complaints about how Lua handles modules. Here and there I see and even answer myself questions regarding require and adjusting the paths in package to allow some desired behaviour, with the most prominent issue of relative imports that always work.

Before we hop into the explanation of how to organize files in your Lua projects, let's talk about default importing mechanism in Lua: require.

How `require` handles paths

Both package and require are surprisingly interesting tools. At first glance they are simple. When you look into them, they are still understandable while gaining some complexity that doesn't reach unnecessary extremes. They are elegant.

They use a mechanism to find desired files called path resolution or usually simply path. The main component of is a sequence of patterns that may become a pathname, e.g.:

/usr/lib/lua/?.lua;/usr/lib/lua/?/init.lua

What does it tell us? First off, ? is going to be replaced by the argument that was provided to the require. All dots will be replaced by an appropriate path separator so that: a.b.c will become a/b/c in *nix systems. So, for call like require "a.b.c", out path will look like this:

/usr/lib/lua/a/b/c.lua;/usr/lib/lua/a/b/c/init.lua

Now, each of these paths are tried and the first one that actually exists in the system will be used. If none of them match an existing file, the import fails. Simple as that.

The path that is used in resolution is set in package.path. You can modify it in Lua, but it is intrusive and may depend on a single entry point. Generally, if you plan to release your project as a module for people to use, I encourage you to avoid modifying anything global. And that's global. Anyway, package.path doesn't appear out of nowhere - it is populated by one of:

Environmental variable LUA_PATH_x_x, where x_x is version such as 5_4
Environmental variable LUA_PATH
Default as defined in luaconf.h

Interestingly, if two separators ;; show up in the environmental variable path, they will be replaced by the default path. Meaning /path/to/project/?.lua;; works as prepending your custom path to the default one.

Of course, there is way more to it than just this i.a.: requiring modules written in C, searchers or preloads. However, in our case this knowledge will suffice.

If you are curious how exactly path is loaded be sure to check out setpath.

Endgame

To prepare for development, we need to know where we are heading. First step is to consider the execution environment. Of course, this and packaging are journeys on their own, so let's just look at two common examples: an application that uses some framework that uses Lua (e.g. a game made in LÖVE) and a standalone module for others to use.

In the first case, it's the duty of the framework to configure the path properly and inform you through the documentation about it. Paths in LÖVE use their own file hierarchy that is managed by love.filesystem and by default contains both the game's source (directory or the mounted .love archive) and the save directory. This means that the structure in your source files is directly reflected in the calls to require, so that require "module.submodule" will always try game/module/submodule.lua, no matter how you run the game. This case usually doesn't involve any additional environment configuration for the development stage.

In the second case, your project will end up in an already configured environment and will need to fit in. The installation of the package usually involves copying your files to the directory that is already included in the path, so that no further configuration is needed for the execution (at least regarding the path). You can assume, that the successful installation will make your modules available in the way you want them.

This doesn't happen in the development stage, when you rarely install your package, and most certainly you don't install it each time you want to test it. This means, that you need to adjust the path so that your modules appear in it as if they were installed in the system. The principle of minimizing the intrusiveness remains, so the best option is to use the environmental variables to prepare for development. If you run your application or any tool in such environment, then Lua will have access to your modules no matter where it is run. Additionally, it will be consistent with the target environment and won't need any additional hacks.

Development environment

All this talk comes down to: set LUA_PATH in your development environment so that it includes your project files even if they are not installed in system. A simple approach is to source following in each session:

export PROJECT=/path/to/project
export LUA_PATH="$PROJECT/?.lua;$PROJECT/?/init.lua;;"

Note the double semicolon that will get replaced by default path, so that other modules that are already installed are also available.

Let's try it out:

$ source env.sh
$ find .
./env.sh
./modulea/submodule.lua
./modulea/init.lua
./moduleb.lua
$ cd modulea
$ lua
Lua 5.4.2
> require "moduleb"
table: 0x561e4b72fb20
> require "modulea"
table: 0x561e4b73aa40
> require "modulea.submodule"
table: 0x561e4b743030

As you can see, despite being in the subdirectory, you can still use modules with their fully qualified names that will remain the same once you install the package. Note, that you could require "init" or require "submodule" in this case, but I strongly recommend against it. Remain specific, follow the rules and pretend that you use an installed package from an unknown working directory. Don't depend on current working directory as it is not always the same. Using full names that consider the path setup guarantees results.

Organizing your files

Finally, this is what we're waiting for. Assume you have a directory that is a parent of all of your project files. We'll call it a project root. Usually, this is also root directory for your version control system, be it git or anything else, and for other tools such as building systems or even entire IDEs.

Because it is such a central place to the project, I usually just go ahead and prepend it to LUA_PATH in the very same way as in the section above:

export PROJECT=/path/to/project
export LUA_PATH="$PROJECT/?.lua;$PROJECT/?/init.lua;;"

Just like previously, any Lua file that will be descendent of the root will be accessible to us through require. But what is that init.lua?

It's there to create a way to improve hierarchical structure of your project - to allow splitting bigger modules into smaller parts (or even submodules that could be included on their own), so that the module doesn't grow into a single millions-lines-long file. In simpler words: you can create a directory named after module and put init.lua file there and it will act just like a sole module.lua in root.

You could also create a directory named after module and module.lua file in root at the same time, but this way you will have two entries per module in the root instead of just one.

Additionally, you can then put any module-related files into that directory. You can also use init.lua as a simple wrapper that calls require for each of its submodules and returns a table with them.

Consider a verbose example:

$ find .
./conf.lua
./env.sh
./main.lua
./persistence/init.lua
./persistence/tests.lua
./version.lua
./wave/init.lua
./wave/sawtooth.lua
./wave/sine.lua
./wave/square.lua
$ cat wave/init.lua
-- This is a wrapper example.
return {
	sawtooth = require "wave.sawtooth",
	sine = require "wave.sine",
	square = require "wave.square",
}
$ cat persistence/init.lua
-- This is a normal module example.
return {}
$ cat persistance/tests.lua
-- This is a script that tests an example module.
local p = require "persistence"
assert(type(p) == "table")
$ cat main.lua
-- This is an example main of love application.
local persistence = require "persistence"
local wave = require "wave"
$ cat version.lua
-- This is an example module that acts as version string of the application.
return "1.0.0"

Now, this is a mash-up of everything we've discussed. Despite it pretending to be LÖVE application it has env.sh. Why? The reason is simple: the persistence and wave modules are not meant to be distributed alone, and they won't ever appear in path of any other environment than LÖVE's. But LÖVE is not the only execution environment in here: persistence/tests.lua is also meant to be executed. Possibly alone through Lua interpreter. To allow it env.sh is present and used.

Let's have another example of a simple module meant for installation:

$ find .
./env.sh
./hello/Class.lua
./hello/init.lua
./hello/tests.lua
./hello/version.lua
./LICENSE
./Makefile
./README
$ cat Makefile
PREFIX?=/usr/local/lib/lua/5.4
all:
	@echo Nothing to be done
install:
	cp -r hello $(PREFIX)
uninstall:
	rm -fd $(PREFIX)/hello/* $(PREFIX)/hello

As you can see Makefile in this example has targets for installation and removal of the package. The structure again is simple. Root works as part of the resolution path and so our module is placed in it's own directory named after it.

The last example is a project of a single file module:

$ find .
./env.sh
./LICENSE
./object.lua
./README

Yes, it's that simple.

Now, having env.sh in every single project might get bothersome, so I usually use a shell function for managing them, similarly to what Python's venv does or LuaRocks' env. Speaking of, LuaRocks is yet another interesting story to be told.

Summary

Consider the execution environment when preparing the development environment.
Contain your project. Avoid changing globals if the modules may be used by someone else in an unknown environment.
Never depend on the current working directory being something specific.
While developing include your project's root in LUA_PATH to influence require's behaviour.
Split modules into parts and use init.lua which is available by default to enclose the module into a single directory.

Alternatives

This is just one of the ways to handle structuring your Lua project. It's based on simple rules but has broad usage. One tempting alternative is this little snippet:

local parents = (...):match "(.-)[^%.]+$"
require(parents .. "sibling")

Another already mentioned alternatives is adjusting package.path directly in Lua. However, I decided to skip it due to it's intrusiveness.

All in all, Lua is extremely customizable and adjustable. I would be surprised if these three would be the only ways to organize projects in Lua.