Illustration
Here we attempt to build a build system in Guix for Elixir.

The objective is to implement a Guix build system for Elixir packages. Success is achieved if, given a correct Guix package definition for any package pkg on Hex, we have guix install elixir-pkg working as expected.

  • The mix build system has been updated in these Guix commits
  • The rebar build system has been updated in these Guix commits

Since phoenix depends on Erlang packages and is one of the most popular Elixir package, we test our implementation using:

guix shell --container --network elixir elixir-phoenix mix phx.new --no-ector hello mix phx.server
  • The Elixir build system is called mix.
  • mix is an executable installed along with an Elixir installation.
  • One of the functions of mix is to build an object code executable by the EVM from Elixir projects.
  • This function is implemented by the mix compile command.
  • Another function is to test an Elixir project.
  • This function is implemented by the mix test command.

phoenix 1.7 depends on telemetry 1.2.1.

telemetry is an Erlang package:

$ mix deps | rg telemetry * telemetry 1.2.1 (Hex package) (rebar3) locked at 1.2.1 (telemetry) dad9ce9d

Since Elixir packages may depend on Erlang packages, an Erlang build system is necessary to implement the Elixir build system.

See: Guix and Erlang Build System

A test Elixir project that depends on the Erlang telemetry package has been built: test_project.tar.gz.

test_project$ iex -S mix ===> Analyzing applications... ===> Compiling telemetry Compiling 3 files (.ex) Generated test_project app Erlang/OTP 26 [erts-14.2.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:ns] Interactive Elixir (1.16.0) - press Ctrl+C to exit (type h() ENTER for help) iex(1)> TestProject.hello("Joe") 12:13:14.944 [info] [Name telemetry: Joe] "Hello Joe!" test_project$ mix test ===> Analyzing applications... ===> Compiling telemetry Compiling 3 files (.ex) Generated test_project app 12:15:30.311 [info] [Name telemetry: Joe] . Finished in 0.01 seconds (0.00s async, 0.01s sync) 1 test, 0 failures
  • We built this project using:
    1. mix new test_project
    2. {:telemetry, "~> 1.2"} is added to mix.exs
    3. Add enough code to show that Elixir uses the Telemetry package.

A Guix package definition has been built for the test project. If this builds, then at least one Elixir package may depend on an Erlang package.

(use-modules (guix packages) (guix download) (guix build-system mix) (guix build-system rebar) (guix gexp) ((guix licenses) #:prefix license:)) (define erlang-telemetry (package (name "erlang-telemetry") (version "1.2.1") (source (origin (method url-fetch) (uri (hexpm-uri "telemetry" version)) (sha256 (base32 "1mgyx9zw92g6w8fp9pblm3b0bghwxwwcbslrixq23ipzisfwxnfs")))) (build-system rebar-build-system) (synopsis "Dynamic dispatching library for metrics and instrumentation") (description "Dynamic dispatching library for metrics and instrumentation.") (home-page "https://hexdocs.pm/telemetry/") (license license:asl2.0))) (package (name "elixir-test-project") (version "0") (source (local-file (canonicalize-path "test_project.tar.gz"))) (inputs (list erlang-telemetry)) (build-system mix-build-system) (synopsis "Test package") (description "Test package.") (home-page "https://example.com") (license #f))

As expected, the build fails:

$ guix build -K -f test-package.scm … ** (Mix) Could not start application telemetry: could not find application file: telemetry.app error: in phase 'check': uncaught exception: %exception #<&invoke-error program: "mix" arguments: ("do" "compile" "--no-deps-check" "--no-prune-code-paths" "+" "test" "--no-deps-check") exit-status: 1 term-signal: #f stop-signal: #f> phase `check' failed after 0.6 seconds command "mix" "do" "compile" "--no-deps-check" "--no-prune-code-paths" "+" "test" "--no-deps-check" failed with status 1 …
  • The build directory has been kept.
    • /tmp/guix-build-elixir-test-project-0.drv-3
  • The telemetry package exists in the environment.
    • Looking at $BUILD_DIR/environment-variables, we find:
      • /gnu/store/…-erlang-telemetry-1.2.1/*

The telemetry app exists:

$ tree /gnu/store/…-erlang-telemetry-1.2.1/ ├── lib │   └── erlang │   └── lib │   └── telemetry │   └── ebin │   ├── telemetry.app │   ├── telemetry_app.beam │   ├── telemetry.beam │   ├── telemetry_handler_table.beam │   ├── telemetry_sup.beam │   └── telemetry_test.beam └── share └── doc └── erlang-telemetry-1.2.1 └── LICENSE

So, mix should be told to look at:

/gnu/store/…-erlang-telemetry-1.2.1/lib/erlang/lib/

The EVM searches for library directories provided by the OS using ERL_LIBS variable. For instance:

ERL_LIBS=/gnu/store/…-erlang-telemetry-1.2.1/lib/erlang/lib

would make the telemetry application available to Elixir. This means that the build systems would have gathered all Erlang dependencies specified in the package inputs to add them to ERL_LIBS.

It builds successfully:

$ guix build -K -f src/test-package.scm […] /gnu/store/fks61451l1wx2gbwzfrd4a7knfyi5m14-elixir-test-project-0

It is the result of this modification of test-package.scm thanks to Hartmut:

(arguments `(#:modules ((guix build mix-build-system) (guix build utils) (ice-9 match)) #:phases (modify-phases %standard-phases (add-before 'build 'collect-erlang-packages (lambda* (#:key inputs #:allow-other-keys) (let* ((erlang-dependencies (filter (match-lambda ((label . _) (string-prefix? "erlang-" label))) inputs)) (erl-libs (string-join (map (match-lambda ((_ . dir) (pk( string-append dir "/lib/erlang/lib")))) erlang-dependencies) ":"))) (setenv "ERL_LIBS" erl-libs)))))))

This piece of code searches for Erlang dependencies in the inputs field of the package and adds the relevant directories to ERL_LIBS so that necessary applications are found at runtime by the EVM.

Moving forward, at least two plans may be followed: using ERL_LIBS or patching the Application Resource File. We choose to follow the ERL_LIBS plan in this article and the Application Resource File plan in an other article.

See: Application resource file plan.

Since:

  • We know that using ERL_LIBS may work.
  • We know that Python uses a similar mechanism with PYTHONPATH.

Then:

  • We study how Python manages its dependencies.
  • We try to apply this knowledge to Elixir.

Counting on the analogy GUIX_ERL_LIBS ~ GUIX_PYTHONPATH, let's find how the value of GUIX_PYTHONPATH is used ; hoping to use GUIX_ERL_LIBS similarly. We have:

python-build-system

For packages that install stand-alone Python programs under bin/, it takes care of wrapping these programs so that their GUIX_PYTHONPATH environment variable points to all the Python libraries they depend on.

In gnu/packages/python.scm, we find:

(define* (customize-site version) "Generate a install-sitecustomize.py phase, using VERSION." `(lambda* (#:key native-inputs inputs outputs #:allow-other-keys) (let* ((out (assoc-ref outputs "out")) (site-packages (string-append out "/lib/python" ,(version-major+minor version) "/site-packages")) (sitecustomize.py (assoc-ref (or native-inputs inputs) "sitecustomize.py")) (dest (string-append site-packages "/sitecustomize.py"))) (mkdir-p site-packages) (copy-file sitecustomize.py dest) ;; Set the correct permissions on the installed file, else the byte ;; compilation phase fails with a permission denied error. (chmod dest #o644))))

In gnu/packages/aux-files/python/sitecustomize.py, we find:

# Site-specific customization for Guix.
#
# The program below honors the GUIX_PYTHONPATH environment variable to
# discover Python packages.  File names appearing in this variable that match
# a predefined versioned installation prefix are added to the sys.path.  To be
# considered, a Python package must be installed under the
# 'lib/pythonX.Y/site-packages' directory, where X and Y are the major and
# minor version numbers of the Python interpreter.
          

The code looks for the value of the environment variable GUIX_PYTHONPATH. This variable may be viewed as the set of transitive Python dependencies of the current package. Each dependency then made available to the Python runtime during initialization.

Counting on the analogy GUIX_ERL_LIBS ~ GUIX_PYTHONPATH, let's find how the value of GUIX_PYTHONPATH is computed ; hoping to apply a similar computation to GUIX_ERL_LIBS. In guix/gnu/packages/python.scm, we find:

(define-module (gnu packages python) ;; … #:export (customize-site guix-pythonpath-search-path)) ;; … (define (guix-pythonpath-search-path version) "Generate a GUIX_PYTHONPATH search path specification, using VERSION." (search-path-specification (variable "GUIX_PYTHONPATH") (files (list (string-append "lib/python" (version-major+minor version) "/site-packages"))))) ;; … (define-public python-3.10 (package ;; … (native-search-paths (list (guix-pythonpath-search-path version) ;; Used to locate tzdata by the zoneinfo module introduced in ;; Python 3.9. (search-path-specification (variable "PYTHONTZPATH") (files (list "share/zoneinfo")))))))
  1. The python-3.10 package defines the GUIX_PYTHONPATH environment variable so that it lists all the Python dependencies in the environment (Guix doc).
  2. Python initialization is modified using sitecustomize.py so that all dependencies represented by GUIX_PYTHONPATH are available to the Python runtime.

To make Erlang/Elixir libraries — i.e. applications — available to the EVM, their directories must be added to the code path. To add them to the code path, they may be listed using the ERL_LIBS environment variable.

The code module contains a number of functions for modifying and querying the search path.

In interactive mode, which is default, only the modules needed by the runtime system are loaded during system startup. Other code is dynamically loaded when first referenced. When a call to a function in a certain module is made, and that module is not loaded, the code server searches for and tries to load that module.

In interactive mode, the code server maintains a code path, consisting of a list of directories, which it searches sequentially when trying to load a module.

Environment variable ERL_LIBS (defined in the operating system) can be used to define more library directories to be handled in the same way as the standard OTP library directory described above, except that directories without an ebin directory are ignored.

All application directories found in the additional directories appear before the standard OTP applications, except for the Kernel and STDLIB applications, which are placed before any additional applications. In other words, modules found in any of the additional library directories override modules with the same name in OTP, except for modules in Kernel and STDLIB.

Any EVM application packaged with Guix — compiled from Elixir or Erlang — can use Elixir or Erlang dependencies, if:

  1. The Erlang package defines the GUIX_ERL_LIBS environment variable so that it lists all the Erlang and Elixir dependencies in the environment.
    • Like Python, we use a search-path-specification.
  2. erl is wrapped so that ERL_LIBS = GUIX_ERL_LIBS.
    • An Erlang runtime system is started with command erl (doc).
  3. The above steps are true for Elixir, except that instead of erl, we have:
    • elixir is wrapped so that ERL_LIBS = GUIX_ERL_LIBS.
    • iex is wrapped so that ERL_LIBS = GUIX_ERL_LIBS.
  4. Elixir and Erlang packages dependencies must be declared as propagated inputs.
    1. GUIX_ERL_LIBS is built using the Search Paths capabilities of Guix.
    2. For a dependency to be found by the Search Paths capabilities of Guix, it must exist in the environment.
    3. For a dependency to exist in the environment, it must be a propagated input.

guix install elixir-phoenix works as expected.