Jenkins Job Naming

We strive for consistency in our Jenkins job naming, so all names are validated using the following PCRE regex:

^(
    ( # define some frequently used capture groups
        (?!x)x                                   # always false ("match only if there is no 'x' ahead and there is 'x' ahead")
        (?<name>[^_]+)                           # group "name", name of the program/library
        (?<os>windows|linux|osx|android|illumos) # group "os", operating systems
        (?<arch>x86-64|x86|arm64|arm)            # group"arch", architectures
        (?<buildtype>release|debug)              # group"buildtype", build types
        (?<linking>shared|static|mixed)          # group "linking", library linking/build types
    )
    |
    ( # regular executable builds
        (?!lib)\g<name>_build_\g<os>_\g<arch>_\g<buildtype>
    )
    |
    ( # library builds
        lib\g<name>_build_\g<os>_\g<arch>_\g<linking>_\g<buildtype>
    )
    |
    ( # syncing source code of a project from git or something else
        \g<name>_src
    )
    |
    ( # static code analizer
        \g<name>_analyze_(cppcheck|scan-build)
    )
    |
    ( # package builds
        \g<name>_pkg_\g<os>_
            (
                ((?<=linux_) # if there is "linux_" before here
                    ( # then match
                        debian-8
                    )_
                )
                |
                ((?<!linux_)) # if there is no "linux_", don't match anything
            )
        \g<arch>_(stable|nightly)_\g<buildtype>
    )
)$

You can test your job name against the regex using this regex101 save.

• libtoxcore_analyze_scan-build
• libtoxcore_build_linux_arm64_mixed_debug
• libtoxcore_build_linux_x86_64_mixed_debug
• libtoxcore_build_windows_x86_shared_release
• libtoxcore_src
• qTox_analyze_cppcheck
• qtox_pkg_linux_debian-8_x86-64_stable_release
• qtox_pkg_windows_x86-64_stable_release
• utox_build_windows_x86_release
• utox_pkg_linux-debian-8_arm_stable_debug
• utox_pkg_osx_x86-64_nightly_debug
• uTox_src

Note that the name part of the job (\g<name>) can contain lower case letters, as well as upper case ones. The general rule of thumb is to use lowercase letters for everything unless you have a good reason to not to. The reason for that rule of thumb is that most jobs initially were named in lowercase only (i.e. libqt5 instead of libQt5, libffmpeg, instead of libFFmpeg, etc.), and sometime later this was enforced by adding lowercase restriction to the regex above, but qTox and uTox maintainers wanted to have “T” letter in their jobs names, so the lowercase-only restriction in regex was lifted because of that, which makes them an exception to the general rule.

If you want to add more rules to the regex or change existing ones, please bring it up for discussion in #tox-dev.

If you will be modifying the regex101 save, be aware that regex101 uses PCRE to evaluate the regex. Jenkins is a Java software, so it uses Java's regex engine, which is not fully compatible with PCRE. For example, Java's regex engine doesn't support backreference constructs \g<name> and conditional constructs (?(condition)X). Read more in Comparison to Perl 5. So even if your regex works fine in regex101, make sure it doesn't use things Java's regex engine doesn't support. Also, be sure to run the unit tests located at the end of the left menu of the regex101 save.

Although Java's regex engine doesn't support backreference constructs \g<name>, you can see us using them in the regex above. We find that construct to be quite useful, since it allows us to change a frequently repeating pattern once and it will change throughout the regex, so we wrote a pre-processor program that translates such multi-line, indented and non-standard commented regular expressions with backreference constructs to a single-line, no indentation, no comments and with backreference constructs replaced by corresponding named capturing group, so that the final regex could be used directly in Jenkins.

It's obvious that such job naming, which might seem a bit too explicit to some, explains right away just what a job is producing. We could have stopped on that fact, since it's quite an achievement by itself, but there is a smart application of it that you can find useful when managing big amount of jobs.

As an example, there are quite a few libqt5_build_* jobs in Jenkins:

1. libqt5_build_linux_x86-64_static_debug
2. libqt5_build_linux_x86-64_static_release
3. libqt5_build_linux_x86_static_debug
4. libqt5_build_linux_x86_static_release
5. libqt5_build_windows_x86-64_shared_debug
6. libqt5_build_windows_x86-64_shared_release
7. libqt5_build_windows_x86-64_static_debug
8. libqt5_build_windows_x86-64_static_release
9. libqt5_build_windows_x86_shared_debug
10. libqt5_build_windows_x86_shared_release
11. libqt5_build_windows_x86_static_debug
12. libqt5_build_windows_x86_static_release

Qt has many build flags that we might need to enable/disable which are almost identical to all of the builds, with just a few differences: static builds pass -static flag, shared builds pass -shared flag, release builds pass -release flag, debug builds pass -debug flag, Linux x86-64 builds pass -platform linux-g++-64 flag, Linux x86 builds pass -platform linux-g++-32 flag and so on, but the majority of flags is the same and if we were to edit some of the flags, it's highly likely to be one of the general flags, so it would need to be changed in all of the libqt5 builds, which are… well, just too many. And if the change didn't succeed, one would need to fix the change, modifying all of the jobs again. It's a pretty cumbersome exercise to do. However, there is a solution to this. If you look closely enough, you will notice that all of the special flags that differentiate one job from another are summarized in the job name. In essence, job name tell us which flags to apply, which we can use to our advantage. Jenkins defines $(JOB_NAME) environment variable (along with many other), which we can use to write a single bash script that will build Qt differently based on nothing but job's name! You can write such a script, make a separate job that produces just that script, make all of libqt5_build_* be downstream projects of that job and make them call the script, so that whenever the script changes, all of libqt5_build_* are rebuilt using the new version of the script.

Here is an example of creating such a script in a Jenkins job using bash's Here Documents, which you can use as a template for creating your own once-script-rules-them-all job:

libqt5-build-script_src

> ${JOB_NAME}.sh cat << "EOF"
#!/bin/bash
 
MAKEFLAGS=j$(nproc)
export MAKEFLAGS
 
mkdir prefix
cd prefix
PREFIX_DIR=$(pwd)
cd ..
 
tar -xf libqt5_src.tar.xz
 
cd qt*
 
# set QT_CONFIGURE_OPTIONS based on ${JOB_NAME}, i.e. OS, architecture, and build types
 
QT_CONFIGURE_OPTIONS="-prefix ${PREFIX_DIR} \
    -opensource -confirm-license \
    -pch \
    -nomake examples \
    -nomake tools \
    -nomake tests \
    -skip translations \
    -skip doc \
    -skip qtdeclarative \
    -skip activeqt \
    -skip qtwebsockets \
    -skip qtscript \
    -skip qtquick1 \
    -skip qtquickcontrols \
    -skip qtconnectivity \
    -skip qtenginio \
    -skip qtsensors \
    -skip qtserialport \
    -skip qtlocation \
    -skip qtgraphicaleffects \
    -skip qttools \
    -skip qtimageformats \
    -skip webkit \
    -skip webkit-examples \
    -skip webchannel \
    -skip multimedia \
    -no-openssl \
    -no-dbus \
    -no-icu \
    -no-qml-debug \
    -qt-libjpeg \
    -qt-libpng \
    -qt-zlib \
    -qt-pcre"
 
if [[ ${JOB_NAME} == *"release"* ]]
then
  QT_CONFIGURE_OPTIONS="-release $QT_CONFIGURE_OPTIONS"
 
elif [[ ${JOB_NAME} == *"debug"* ]]
then
  QT_CONFIGURE_OPTIONS="-debug $QT_CONFIGURE_OPTIONS"
 
else
  echo "Error: Unknown build type, neither release or debug. Exiting."
  exit 1
fi
 
 
if [[ ${JOB_NAME} == *"static"* ]]
then
  QT_CONFIGURE_OPTIONS="-static $QT_CONFIGURE_OPTIONS"
 
elif [[ ${JOB_NAME} == *"shared"* ]]
then
  QT_CONFIGURE_OPTIONS="-shared $QT_CONFIGURE_OPTIONS"
 
else
  echo "Error: Unknown build type, neither static or shared. Exiting."
  exit 1
fi
 
 
if [[ ${JOB_NAME} == *"windows"* ]]
then
 
  if [[ ${JOB_NAME} == *"x86_64"* ]]
  then
    QT_CONFIGURE_OPTIONS="-device-option CROSS_COMPILE=x86_64-w64-mingw32- $QT_CONFIGURE_OPTIONS"
 
  elif [[ ${JOB_NAME} == *"x86"* ]]
  then
    QT_CONFIGURE_OPTIONS="-device-option CROSS_COMPILE=i686-w64-mingw32- $QT_CONFIGURE_OPTIONS"
 
  else
    echo "Error: Support only x86 and x86_64 architectires on Windows. Exiting."
    exit 1
  fi
 
  QT_CONFIGURE_OPTIONS="-xplatform win32-g++ $QT_CONFIGURE_OPTIONS"
 
elif [[ ${JOB_NAME} == *"linux"* ]]
then
 
  if [[ ${JOB_NAME} == *"x86_64"* ]]
  then
    QT_CONFIGURE_OPTIONS="-platform linux-g++-64 $QT_CONFIGURE_OPTIONS"
 
  elif [[ ${JOB_NAME} == *"x86"* ]]
  then
    QT_CONFIGURE_OPTIONS="-platform linux-g++-32 $QT_CONFIGURE_OPTIONS"
 
  else
    echo "Error: Support only x86 and x86_64 architectires on Linux. Exiting."
    exit 1
  fi
 
  QT_CONFIGURE_OPTIONS="$QT_CONFIGURE_OPTIONS -qt-freetype -qt-xcb"
 
fi
 
./configure ${QT_CONFIGURE_OPTIONS}
make
make install
 
cd ..
 
rm -rf qt*
rm libqt5_src.tar.xz
 
cd "${PREFIX_DIR}"
 
if [[ ${JOB_NAME} == *"windows"* ]]
then
  zip ${JOB_NAME}.zip -r *
  mv ${JOB_NAME}.zip ..
elif [[ ${JOB_NAME} == *"linux"* ]]
then
  tar -cJf ${JOB_NAME}.tar.xz *
  mv ${JOB_NAME}.tar.xz ..
fi
 
# break the hardcoded into Qt build path
mv "${PREFIX_DIR}" "${PREFIX_DIR}"_
EOF
 
chmod +x ${JOB_NAME}.sh

Note that ${JOB_NAME} inside and outside of Here Documents is different. Outside it evaluate to the script's job name – “libqt5-build-script_src”, when inside it's written into a file as “${JOB_NAME}”, without being evaluated, i.e. it will be evaluated when someone actually calls the “libqt5-build-script_src.sh” script.

All of the numerous libqt5_build_* jobs now have to do, is simply call ./libqt5-build-script_src.sh and archive the artifact. Also, whenever you modify the script, all of libqt5_build_* are rebuilt with the new version of the script, which is exactly what we want (if not, it's still easy enough to manually cancel the job, a lot easier than manually scheduling them to build).