Preloader: Generator of Lua Module Pre-Loading Code

If you want to embed a Lua script library like LOOP in a C/C++ application, one option is to create a C library with all your library's script files already pre-compiled so it can pre-load them into a lua_State*. This way your entire library becomes available to any script that runs in that lua_State*. To create such library, use the script preloader.lua provided in the LOOP package.

preloader.lua works as a sort of replacement for the old bin2c application that used to be included in the Lua distribution, however it is primarily intended to pre-compile and pre-load scripts that implement modules following the Lua 5.1 package model. To embed ordinary scripts already pre-compiled into C/C++ applications check out bin2c or other implementations of the old bin2c application. On the other hand, preloader.lua is very general and may be used to generate pre-compiled libraries for any set of Lua scripts.

WARNING:

The current version of preloader.lua require class Argument Processor to handle command-line options. So it might be necessary to make the LOOP Class Library available in the LUA_PATH. Alternatively, you may simply redefine your command as:

lua -epackage.path=[[<LOOP_HOME>/?.lua]] <LOOP_HOME>/preloader.lua

Anyway, preloader.lua is very easy to read and modify, so feel free to change it in any way to better fit your demands.

On the other hand, these concerns are not valid if you installed Preloader using LuaRocks. In such case, you can use this script simply by command preloader.lua.

Usage Overview

$ preloader.lua [options] [inputs]

[inputs] is a sequence Lua script files, C header files of Lua libraries or even Lua module names. Use the options described below to indicate how the [inputs] should be interpreted. If no [inputs] are provided then such names are read from the standard input. The [options] accepted are described below:

-b, -bytecodes: Indicates the provided inputs define files containing compiled bytecodes instead of source code, like the files produced by the luac compiler. When this flag is used no compilation is performed by this script.
-c, -compileonly: Disables the generation of a preloader function. This flag implicitly forces the use of flag -modfuncs.
-def: Defines the name of the file to be generated containing the names of the functions that shall be exported.
-d, -directory: Defines the directory where the output files should be generated. The default value is the current directory.
-f, -funcload: Defines the name of the pre-loader function, a function that pre-loads all modules in a Lua state. The default value is luapreload_ plus the name defined by option -output.
-h, -header: Defines the name of the header file to be generated with the signature of all generated functions (module functions and the pre-loader function). If this option is not provided, no header file is generated.
-I, -i, -include: Adds a directory to the list of paths where the C header files are searched.
-l, -luapath: Defines a sequence of path templates used to infer module names from script paths or vice versa. These templates follow the same format of the package.path field of Lua. The default value is the current value of package.path.
-m, -modfuncs: Enables the generation of functions to load the compiled script modules (luaopen_*). These functions follow the format defined by the Lua package model. Thus they can be exported by a dynamic library to load the compiled script modules by the standard Lua runtime.
-n, -names: Flag that indicates the provided input names are actually module names and not paths to script files and C header files. Each module name provided is applied to the path defined by -luapath option to find the file containing the implementation of the module. If no file can be found then the module is assumed to be implemented by a C file that must be linked with the generated file to produce the final binary. This flag can be used in conjunction with the -bytecodes flag to indicate that inferred file paths contains bytecodes instead of source code.
-o, -output: Defines the name of the output file with all generated functions: module functions and the preloader function. The default value is preload.c.
-p, -prefix: Defines the macro added before the signature of the functions generated. The default value is LUAOPEN_API.
-s, -signatures: Disables the use of '#include' directives to include the C header files of Lua libraries processed, generating function signatures instead.
-w, -warnings: Enables the generation of warning messages in the standard error output.

Pre-Loading Script Modules

To generate a library that pre-loads your script modules, first select the files that implement the modules you want to pre-load. Next, call preloader.lua to generate the C source file of the pre-loader library providing as arguments the paths of all script files that shall be pre-loaded. The path to these script files must follow the same hierarchy of the modules that they implement so the module name shall be correctly inferred from the provided -luapath. See below an example of how to use preloader.lua to generate a library that pre-loads all LOOP modules:

$ preloader.lua -output loop.c -luapath "lua/?.lua" \
                lua/loop/base.lua \
                lua/loop/cached.lua \
                lua/loop/classops.lua \
                lua/loop/hierarchy.lua \
                lua/loop/multiple.lua \
                lua/loop/proto.lua \
                lua/loop/scoped/debug.lua \
                lua/loop/scoped.lua \
                lua/loop/simple.lua \
                lua/loop/table.lua

The name of the module implemented by each file is inferred by matching the provided path with the path patterns defined with option -luapath. No path expansion is performed in this matching, so precompiler.lua won't be able to figure out the module name of file mypacks/loop/base.lua if the provided -luapath is /home/user/mypacks/?.lua even though the current directory is /home/user. However if -luapath is set to mypacks/?.lua the inferred module name would be loop.base. When the -luapath defines multiple patterns, the last (i.e. rightmost) that matches the file path is used to define the module name.

Alternatively, you can use the option -names to provide module names instead of script file paths. In such case, the provided -luapath is used to figure out the file that contains the actual module implementation. This is useful when module name cannot be correctly inferred from the file path, like in case-insensitive file systems. Therefore the above example could be rewritten like below:

$ preloader.lua -output loop.c -luapath "lua/?.lua" -names \
                loop.base \
                loop.cached \
                loop.hierarchy \
                loop.multiple \
                loop.proto \
                loop.scoped.debug \
                loop.scoped \
                loop.simple \
                loop.table \
                loop

Additionally, if option -bytecodes is provided then preloader.lua assumes the referred files contain bytecodes instead of source code. This is useful to create a library with scripts pre-compiled with luac for platforms other than the one where the preloader.lua is executing.

Finally, if no additional parameter is provided other than the options listed above, then the file paths or module names are read from the standard input. Therefore, if the list of modules to be compiled is stored in file loop.lpk and they are already compiled in directory compiled in files with extension .lo that follow the same hierarchy of the modules then the command could be:

$ preloader.lua -output loop.c -luapath "compiled/?.lo" -names -bytecodes < loop.lpk

As a result of the commands listed above you will get file loop.c that implements function int luapreload_loop(lua_State*) that you can call to pre-load all these script modules into Lua. You can define the name of this pre-load function with option -funcload.

Additionally, you can use option -header to define the name of a header file to be generated with signatures of all functions generated.

Pre-Loading C Modules

preloader.lua can also be used to create libraries that pre-loads Lua modules written in C. In this case, you provide to preloader.lua the C header files of the C library that implements the Lua module to be pre-loaded instead of script files. For example, to generate a library that pre-loads the full LuaSocket library, including its script files, you can use the following command.

$ preloader.lua -output luasocketlib.c -luapath "?.lua" \
                socket.lua \
                socket/ftp.lua \
                socket/http.lua \
                socket/smtp.lua \
                socket/tp.lua \
                socket/url.lua \
                luasocket.h

In this case, preloader.lua matches each file path with the provided -luapath to figure out if it is a script file or a C header file. Since luasocekt.h does not match in any pattern of the provided -luapath, it assumes it is a header file and searches for any functions declared that loads Lua modules (i.e. luaopen_* functions).

Just like the case with script modules, you can provide the names of the modules that should be pre-loaded instead of header files. To do so, just provide the same option -names to preloader.lua. This is useful when module name cannot be inferred correctly from the luaopen_* function. For example a module named my_lib.my_module is loaded by function luaopen_my_lib_my_module that preloader.lua script would register under the name my.lib.my.module. For example, the command above can be rewritten using names instead of file paths as illustrated below:

$ preloader.lua -output luasocketlib.c -luapath "?.lua" -names \
                socket \
                socket.ftp \
                socket.http \
                socket.smtp \
                socket.tp \
                socket.url \
                socket.core

In this case, preloader.lua uses the provided -luapath to find the script files that implement each module. Whenever it is not able to find the script file, it assumes that the module is provided by a luaopen_* function from another C file or library that will be linked with the generated file luasocketlib.c.

Use option -warnings to see warning messages about the assumptions that preloader.lua makes about its inputs.

As a result of the commands above you will get file luasocketlib.c that shall be compiled with the C source files of the LuaSocket library to produce a library that provides function int luapreload_luasocketlib(lua_State*) that pre-loads into a provided Lua state the full LuaSocket library, including its script files.

Bundle Libraries

preloader.lua can also be used to produce a library that provides functions to load each script module provided as input. The generated functions follow the pattern defined by the Lua package model. Therefore, if these functions are exported by a dynamic library installed in a path specified in the package.cpath then these modules will be automatically available in Lua.

To make preloader.lua generate module loading functions use option -modfuncs. Alternatively, use option -compileonly to suppress the generation of a pre-loader function and only generate module loading functions (i.e. luaopen_* functions). For example, to generate a library with all luaopen_* functions necessary to load LOOP modules, you can use the following command:

$ preloader.lua -output loop.c -header loop.h -luapath "lua/?.lua" -compileonly \
                lua/loop/base.lua \
                lua/loop/cached.lua \
                lua/loop/classops.lua \
                lua/loop/hierarchy.lua \
                lua/loop/multiple.lua \
                lua/loop/proto.lua \
                lua/loop/scoped/debug.lua \
                lua/loop/scoped.lua \
                lua/loop/simple.lua \
                lua/loop/table.lua

As a result of this command, you will get files loop.c and loop.h that provides no pre-loader function, only the following module loading functions:

int luaopen_loop_base(lua_State*);
int luaopen_loop_cached(lua_State*);
int luaopen_loop_classops(lua_State*);
int luaopen_loop_hierarchy(lua_State*);
int luaopen_loop_multiple(lua_State*);
int luaopen_loop_proto(lua_State*);
int luaopen_loop_scoped_debug(lua_State*);
int luaopen_loop_scoped(lua_State*);
int luaopen_loop_simple(lua_State*);
int luaopen_loop_table(lua_State*);

Download

You can install Preloader through LuaRocks using the following command.

   luarocks install preloader

This utility is provided as free software under the terms of the MIT License, which are the same terms of the Lua License.

For download see the list of released packages.

Author

This project was created by Renato Maia.