Guix — Rebar Build System — Implementation

Introduction

Given the prototype (Web), the objective of this post is to provide a Guix implementation and document it for future reference. In particular, the following objectives should be fulfilled:

  • ☐ The particularities of Guix Erlang Packages are defined.
  • ☐ Compiled libraries are installed under lib/erlang/X.Y, where X.Y represents the major and minor version numbers of the Erlang used to compile them.
  • ☐ The Rebar build system aligns with the latest API of rebar3.

Guix Erlang Package

A fictional package with all properties specific to Erlang is given below along with their descriptions:

(package

  ;; name has the form erlang-<libname> where <libname> is the Hex
  ;; package name with "_" replaced by "-".
  (name "erlang-nimble-parsec")

  (version "1.0.2")
  (source
   (origin
     (method url-fetch)
     (uri (hexpm-uri name version))
     (sha256
      (base32 "…"))))
  (arguments
   '(#:tests? #t

     ;; Default to "eunit".  Invoked as `rebar3 <test-target>'.
     #:test-target "eunit"

     ;; Default to "default".  Invoked within `rebar3 as
     ;; <install-profile> compile'.
     #:install-profile "default"))

  ;; List of dependencies, in particular those in rebar.conf/deps.
  (inputs (list ...))

  ;; List of runtime dependencies, in particular those in
  ;; app.src/applications/deps.
  (propagated-inputs (list ...))

  ;; List of test dependencies, in particular those in
  ;; rebar.conf/profiles/test/deps.
  (native-inputs (list ...))

  (build-system rebar-build-system)
  (home-page "…")
  (synopsis "…")
  (description "…")
  (license "…"))

Compilation artifacts

A given Erlang Package is mapped to files in the store (artifacts). The associated paths follow this template:

/gnu/store/<hash>-erlang-<pkgname>-<version>/
├── bin
│   └── …
└── lib
    └── erlang
        └── <X.Y>
            └── <libname>
                ├── ebin
                │   └── …
                ├── include
                │   └── …
                └── priv
                    └── …

Compilation

The standard compilations procedure — as far as the Rebar documentation goes — adapted to Guix is:

  1. Let pkg be an Erlang Guix package.
  2. pkg source code is fetched.
  3. Erlang dependencies are added as checkouts.
  4. rebar3 as default compile is executed.
  5. rebar3 eunit is executed.
  6. Compiled artifacts are installed in the store.

If a package needs a special procedure, then one may use the parameters described above in addition to the standard customization options provided by Guix e.g. #:phases … .

rebar.scm

The objective of rebar.scm is to define rebar-build-system, an instance of a Build System (Web). A Build System is a function : Package → Artifact that can be thought of as follow:

  1. lower : Package → Bag
  2. rebar-build : Bag → Derivation
  3. guix-daemon : Derivation → Artifact

Header

;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2016 Ricardo Wurmus <rekado@elephly.net>
;;; Copyright © 2020 Hartmut Goebel <h.goebel@crazy-compilers.com>
;;; Copyright © 2023 Pierre-Henry Fröhring <contact@phfrohring.com>
;;;
;;; This file is part of GNU Guix.
;;;
;;; GNU Guix is free software; you can redistribute it and/or modify it
;;; under the terms of the GNU General Public License as published by
;;; the Free Software Foundation; either version 3 of the License, or (at
;;; your option) any later version.
;;;
;;; GNU Guix is distributed in the hope that it will be useful, but
;;; WITHOUT ANY WARRANTY; without even the implied warranty of
;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;;; GNU General Public License for more details.
;;;
;;; You should have received a copy of the GNU General Public License
;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.

Modules

(define-module (guix build-system rebar)
  #:use-module (guix store)
  #:use-module (guix utils)
  #:use-module (guix gexp)
  #:use-module (guix packages)
  #:use-module (guix monads)
  #:use-module (guix search-paths)
  #:use-module (guix build-system)
  #:use-module (guix build-system gnu)
  #:use-module (srfi srfi-26)
  #:export (hexpm-uri
            hexpm-package-url
            %rebar-build-system-modules
            rebar-build
            rebar-build-system))

hex.pm

  • hex.pm is the primary repository of packages for the Erlang ecosystem.
  • An API that abstracts details of the Hex API is offered to the packager.
;; See: https://github.com/hexpm/specifications/blob/master/endpoints.md
(define %hexpm-repo-url
  (make-parameter "https://repo.hex.pm"))

(define hexpm-package-url
  (string-append (%hexpm-repo-url) "/tarballs/"))

(define* (strip-prefix name #:optional (prefix "erlang-"))
  "Return NAME without the prefix PREFIX."
  (if (string-prefix? prefix name)
      (string-drop name (string-length prefix))
      name))

(define (hexpm-uri name version)
  "Return the URI where to fetch the sources of a Hex package NAME at VERSION.
NAME is the name of the package which should look like: erlang-pkg-name-X.Y.Z
See: https://github.com/hexpm/specifications/blob/main/endpoints.md"
  ((compose
    (cute string-append "https://repo.hex.pm/tarballs/" <> "-" version ".tar")
    (cute string-replace-substring <> "-" "_"))
   (strip-prefix name)))

lower

  • A bag is built by adding implicit dependencies to the package definition.
  • Rebar is added.
  • Erlang is added.
  • GNU standard packages are added.
  • Since cross compilation is not supported (yet):
    • An error is thrown if target is specified.
    • host-inputs is the empty list.
(define (default-rebar3)
  "Return the default Rebar3 package."
  ;; Lazily resolve the binding to avoid a circular dependency.
  (let ((erlang-mod (resolve-interface '(gnu packages erlang))))
    (module-ref erlang-mod 'rebar3)))

(define (default-erlang)
  "Return the default Erlang package."
  ;; Lazily resolve the binding to avoid a circular dependency.
  (let ((erlang-mod (resolve-interface '(gnu packages erlang))))
    (module-ref erlang-mod 'erlang)))

(define* (lower name
                #:key source inputs native-inputs outputs system target
                (rebar (default-rebar3))
                (erlang (default-erlang))
                #:allow-other-keys
                #:rest arguments)
  "Return a bag for NAME from the given arguments."

  (when target
    (error (format #f "Cross compilation is not supported by the Rebar Build System
and a target has been specified: ~a" target)))

  (bag
    (name name)
    (system system)
    (build-inputs `(,@(standard-packages)
                    ("erlang" ,erlang)
                    ("rebar" ,rebar)
                    ,@inputs
                    ,@native-inputs))
    (host-inputs '())
    (outputs outputs)
    (arguments (strip-keyword-arguments
                '(#:target #:rebar #:erlang #:inputs #:native-inputs)
                arguments))
    (build rebar-build)))

rebar-build

  • Given a bag, a derivation is built.
  • The derivation represents instructions to be executed by the Guix daemon to produce the package artifacts.
  • To understand the following code, G-Expressions must be understood.
  • Whatever comes after #:key correspond to the arguments of the bag.
  • Default values are added to the arguments.
  • The parameters of the derivation are set here.
  • The instructions of the derivation are defined in rebar-build in rebar-build-system.scm.
(define %rebar-build-system-modules
  ;; Build-side modules imported by default.
  `((guix build rebar-build-system)
    ,@%gnu-build-system-modules))

(define* (rebar-build name inputs
                      #:key
                      guile
                      source
                      (tests? #t)
                      (test-target "eunit")
                      (install-profile "default")
                      (phases '(@ (guix build rebar-build-system) %standard-phases))
                      (outputs '("out"))
                      (search-paths '())
                      (native-search-paths '())
                      (system (%current-system))
                      (imported-modules %rebar-build-system-modules)
                      (modules '((guix build rebar-build-system)
                                 (guix build utils))))
  "Build SOURCE with INPUTS."

  (define builder
    (with-imported-modules imported-modules
      #~(begin
          (use-modules #$@(sexp->gexp modules))
          #$(with-build-variables inputs outputs
              #~(rebar-build #:source #+source
                             #:system #$system
                             #:name #$name
                             #:tests? #$tests?
                             #:test-target #$test-target
                             #:install-profile #$install-profile
                             #:phases #$(if (pair? phases)
                                            (sexp->gexp phases)
                                            phases)
                             #:outputs %outputs
                             #:search-paths '#$(sexp->gexp
                                                (map search-path-specification->sexp
                                                     search-paths))
                             #:inputs %build-inputs)))))

  (mlet %store-monad ((guile (package->derivation (or guile (default-guile))
                                                  system #:graft? #f)))
    ;; Note: Always pass #:graft? #f.  Without it, ALLOWED-REFERENCES &
    ;; co. would be interpreted as referring to grafted packages.
    (gexp->derivation name builder
                      #:system system
                      #:target #f
                      #:graft? #f
                      #:guile-for-build guile)))

rebar-build-system

(define rebar-build-system
  (build-system
    (name 'rebar)
    (description "The standard Rebar build system")
    (lower lower)))

rebar-build-system.scm

  • The objective of this file is to define instructions to be executed by the Guix daemon to build packages artifacts.

Header

;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2016, 2018 Ricardo Wurmus <rekado@elephly.net>
;;; Copyright © 2019 Björn Höfling <bjoern.hoefling@bjoernhoefling.de>
;;; Copyright © 2020, 2022 Hartmut Goebel <h.goebel@crazy-compilers.com>
;;; Copyright © 2023 Pierre-Henry Fröhring <contact@phfrohring.com>
;;;
;;; This file is part of GNU Guix.
;;;
;;; GNU Guix is free software; you can redistribute it and/or modify it
;;; under the terms of the GNU General Public License as published by
;;; the Free Software Foundation; either version 3 of the License, or (at
;;; your option) any later version.
;;;
;;; GNU Guix is distributed in the hope that it will be useful, but
;;; WITHOUT ANY WARRANTY; without even the implied warranty of
;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;;; GNU General Public License for more details.
;;;
;;; You should have received a copy of the GNU General Public License
;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.

Modules

(define-module (guix build rebar-build-system)
  #:use-module ((guix build gnu-build-system) #:prefix gnu:)
  #:use-module ((guix build utils) #:hide (delete))
  #:use-module (ice-9 match)
  #:use-module (ice-9 ftw)
  #:use-module (srfi srfi-1)
  #:use-module (srfi srfi-26)
  #:export (rebar-build
            %standard-phases))

unpack

(define* (unpack #:key source #:allow-other-keys)
  "Unpack SOURCE in the working directory, and change directory within the
source.  When SOURCE is a directory, copy it in a sub-directory of the current
working directory."
  (let ((gnu-unpack (assoc-ref gnu:%standard-phases 'unpack)))
    (gnu-unpack #:source source)
    ;; Packages from hex.pm typically have a contents.tar.gz containing the
    ;; actual source. If this tar file exists, extract it.
    (when (file-exists? "contents.tar.gz")
      (invoke "tar" "xvf" "contents.tar.gz"))))

set-rebar-env

(define* (set-rebar-env #:key inputs #:allow-other-keys)
  "Set environment variables."
  (setenv "REBAR_CACHE_DIR" (getcwd))
  (setenv "REBAR_GLOBAL_CONFIG_DIR" (getcwd))
  (setenv "REBAR_BASE_DIR" "_build")
  (setenv "REBAR_CHECKOUTS_DIR" "_checkouts"))

set-erlang-version

  • Since compiled libraries depend on the version of Erlang used to compile them, we need to find the major and minor version numbers — X.Y — of the Erlang used to compile them to later write them under a directory of the form lib/erlang/X.Y.
(define* (strip-prefix name #:optional (prefix "erlang-"))
  "Return NAME without the prefix PREFIX."
  (if (string-prefix? prefix name)
      (string-drop name (string-length prefix))
      name))

;; The Erlang version is constant as soon as it is computable from the current
;; execution.  It is a X.Y string where X and Y are respectively the major and
;; minor version number of the Erlang used in the build.
(define %erlang-version (make-parameter "X.Y"))

(define (erlang-version inputs)
  "Return an X.Y string where X and Y are respectively the major and
minor version number of PACKAGE. Example: /gnu/store/…-erlang-1.14.0 →
1.14"
  ((compose
    (cute string-join <> ".")
    (cute take <> 2)
    (cute string-split <> #\.)
    strip-prefix
    strip-store-file-name)
   (assoc-ref inputs "erlang")))

(define* (set-erlang-version #:key inputs #:allow-other-keys)
  "Store the version number of the Erlang input in a parameter."
  (%erlang-version (erlang-version inputs))
  (format #t "Erlang version: ~a~%" (%erlang-version)))

add-dependencies

  • Rebar will refuse to compile anything if the dependencies are not located in a _checkouts directory.
(define (list-directories dir)
  "List absolute paths of directories directly under the directory DIR."
  (scandir dir (lambda (filename)
                 (and (not (member filename '("." "..")))
                      (directory-exists? (string-append dir "/" filename))))))

(define* (erlang-libdir path #:optional (version (%erlang-version)))
  "Return the path where all libraries under PATH for a specified Erlang
VERSION are installed."
  (string-append path "/lib/erlang/" version))

(define* (add-dependencies #:key inputs profiles #:allow-other-keys)
  (define input-directories
    (match inputs (((_ . dir) ...) dir)))

  (define checkouts (getenv "REBAR_CHECKOUTS_DIR"))
  (mkdir-p checkouts)

  ;; For all libname in elibdir, elibdir/libname → _checkouts/libname.
  (define (install-libs elibdir checkouts)
    (for-each (lambda (libname)
                (let ((dst (string-append checkouts "/" libname)))
                  (when (not (file-exists? dst))
                    (symlink (string-append elibdir "/" libname) dst))))
              (list-directories elibdir)))

  ;; For all elibdir in inputs, (install-libs elibdir checkouts).
  (for-each (lambda (input-dir)
              (let ((elibdir (erlang-libdir input-dir)))
                (when (directory-exists? elibdir)
                  (install-libs elibdir checkouts))))
            input-directories))

build

(define* (build #:key install-profile #:allow-other-keys)
  (invoke "rebar3" "as" install-profile "compile"))

check

(define* (check #:key tests? test-target #:allow-other-keys)
  (if tests?
      (invoke "rebar3" test-target)
      (format #t "test suite not run~%")))

install

  • Finally, the compiled libraries and binaries are added to the output directory.
(define (package-name->erlang-name name+ver)
  "Convert the Guix package NAME-VER to the corresponding Elixir name-version
format.  Example: erlang-a-pkg-1.2.3 -> a_pkg"
  ((compose
    (cute string-join <> "_")
    (cute drop-right <> 1)
    (cute string-split <> #\-))
   (strip-prefix name+ver)))

(define* (install #:key name outputs install-profile #:allow-other-keys)
  "Install artifacts in the store.
Copy _build/<profile>/bin under out/.
Copy directories _build/<profile>/lib/{ebin,priv} under <out>/lib/erlang/<X.Y>/<libname>/.
"
  (let* ((out (assoc-ref outputs "out"))
         (install-dir (string-append (getenv "REBAR_BASE_DIR") "/" install-profile))
         (install-name (package-name->erlang-name name))
         (bin-dir (string-append install-dir "/bin"))
         (lib-dir (string-append install-dir "/lib/" install-name))
         (lib-dir-out (string-append (erlang-libdir out) "/" install-name)))

    (when (file-exists? bin-dir)
      (copy-recursively bin-dir out #:follow-symlinks? #t))

    (mkdir-p lib-dir-out)
    (for-each
     (lambda (dirname)
       (let ((src (string-append lib-dir "/" dirname))
             (dst (string-append lib-dir-out "/" dirname)))
         (when (file-exists? src)
           (mkdir-p dst)
           (copy-recursively src dst #:follow-symlinks? #t))))
     '("ebin" "priv"))))

%standard-phases

(define %standard-phases
  (modify-phases gnu:%standard-phases
    (replace 'unpack unpack)
    (add-after 'unpack 'set-rebar-env set-rebar-env)
    (add-after 'set-rebar-env 'set-erlang-version set-erlang-version)
    (delete 'bootstrap)
    (delete 'configure)
    (add-before 'build 'add-dependencies add-dependencies)
    (replace 'build build)
    (replace 'check check)
    (replace 'install install)))

rebar-build

(define* (rebar-build #:key inputs (phases %standard-phases)
                      #:allow-other-keys #:rest args)
  "Build the given Erlang package, applying all of PHASES in order."
  (apply gnu:gnu-build #:inputs inputs #:phases phases args))

All Guix Erlang Packages build

This script is executed without errors:

#! /usr/bin/env bash

set -euo pipefail
IFS=$'\n\t'

PACKAGES=(
  erlang-bbmustache
  erlang-certifi
  erlang-cf
  erlang-covertool
  erlang-cth-readable
  erlang-edown
  erlang-erlware-commons
  erlang-eunit-formatters
  erlang-getopt
  erlang-hex-core
  erlang-jsone
  erlang-jsx
  erlang-parse-trans
  erlang-proper
  erlang-providers
  erlang-relx
  erlang-ssl-verify-fun
  erlang-yamerl
)

for pkg in "${PACKAGES[@]}"; do
  ./pre-inst-env guix build "$pkg"
done

Conclusion

We have defined the Guix Erlang packages and implemented rebar.scm and rebar-build-system.scm so that all Guix Erlang packages build.