                    Finite Element Discretization Library
                                   __
                       _ __ ___   / _|  ___  _ __ ___
                      | '_ ` _ \ | |_  / _ \| '_ ` _ \
                      | | | | | ||  _||  __/| | | | | |
                      |_| |_| |_||_|   \___||_| |_| |_|

                               https://mfem.org


This directory contains most of the GitLab CI configuration. MFEM runs both PR
and nightly testing on GitLab.

# Structure

## Top level

The root configuration file is `.gitlab-ci.yml` at the root of MFEM repo. This
file only defines three stages, a prerequisites one, and two main stages in
which we trigger several sub-pipelines.

We use sub-pipelines to isolate the test for one combination of `machine`
and `test type`.

Machines typically include:

* Dane: Intel Sapphire Rapids
* Matrix: Intel Sapphire Rapids + Nvidia H100 GPU
* Tioga: AMD MI250X GPU

Test types include:

* Build and test: Spack driven build of dependencies, mfem build, mfem
  test
* Baseline: Script driven build of dependencies, thorough testing

⚠️ The sub-pipeline design allows to add a new machine or a new test type without
altering the scheduling, execution and displaying of the others.

## Sub-pipelines

### build-and-test

The build-and-test sub-pipelines leverage RADIUSS Shared CI to share most of
the CI implementation. RADIUSS Shared CI provides a shared CI infrastructure
vetted on most LC systems of interest and efficiently leveraging each machine
scheduler to increase CI throughput. The maintenance of RADIUSS Shared CI is
shared among several RADIUSS projects.

Jobs for the build-and-test sub-pipelines are defined in the jobs directory.
Because build-and-test jobs leverage Uberenv and Spack to build the
dependencies automatically, the jobs essentially consists in a `spack spec`
defined in the jobs files, and some scheduling parameters defined in the
`.gitlab/custom-jobs-and-variables.yml` file.

Build-and-test jobs all run the `tests/gitlab/build_and_test` script.

The build-and-test pipelines are controlled by the
`.gitlab/subscribed-pipelines.yml` which defines which machines to run on and
implements additional features like machine availability check, and job list
generation.

### baseline

Baseline sub-pipelines are described by files with names reflecting the
machine it runs on, e.g. `dane-baseline`.

Those files define the *stages* and the *jobs* for the sub-pipeline. They
also contain any configuration that cannot be shared. For the most part
though, the configuration is shared and is placed in `.gitlab/configs`.

We try to keep scripts out of the CI config and share them among similar
jobs. They are gathered in `.gitlab/scripts`.

## Scripts

Scripts specific to the CI only are in `.gitlab/scripts`. It is best practice
to keep scripts outside the CI configuration (no bash scripts embedded in a
yaml file) because it helps with readability, maintenance and also with
transition to another CI system.

⚠️ Most of the scripts there are driven by environment variables and do not have a
usage function. This should be improved.


# More testing

## Adding a new target to a build-and-test pipeline

`build-and-test` pipelines rely on Spack to install dependencies. Spack is
driven by Uberenv which helps freezing Spack configuration: the goal being to
point to a specific commit in Spack and isolate its configuration so that it is
not influenced by the user environment. More documentation about this can be
found in `tests/gitlab`.

In the end, the MFEM target for which to build the dependencies is expressed
with a spack spec of MFEM, within the limits permitted by the MFEM spack
package.

In any build-and-test sub-pipeline a job basically consists in defining the
spack spec to use. Adding a job on Dane for example resumes to:

```yaml
<job_name>:
  variables:
    SPEC: "<spack_spec>"
  extends: .job_on_dane
```

The remaining and non trivial work is to make sure this spec is working. To
test a spec before adding it, or reproduce a CI configuration, please refer to
`tests/gitlab/reproduce-ci-jobs-interactively.md`.

⚠️ It is assumed that the spack spec applies to `mfem@develop`. That's why in the
CI all the specs start with the compiler or the variants to apply to mfem. The
mechanism still works with a full spec.
