Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
developers:jenkins:job_naming [2015/08/17 02:47] – updated regex, added a few paragraphs explaining how it should be modified nurupodevelopers:jenkins:job_naming [2017/10/08 07:57] (current) – [Jenkins Job Naming] Added Ubuntu Artful to the regex nurupo
Line 6: Line 6:
 ^( ^(
     ( # define some frequently used capture groups     ( # define some frequently used capture groups
-        (?!x)x                                   # always false ("match only if there is no 'x' ahead and there is 'x' ahead") +        (?!x)x                                          # always false ("match only if there is no 'x' ahead and there is 'x' ahead") 
-        (?<name>[^_]+)                           # group "name", name of the program/library +        (?<name>[^_]+)                                  # group "name", name of the program/library 
-        (?<os>windows|linux|osx|android|illumos) # group "os", operating systems +        (?<os>windows|linux|osx|android)                # group "os", operating systems 
-        (?<arch>x86-64|x86|arm64|arm)            # group"arch", architectures +        (?<arch>x86-64|x86|arm64|armhf|armel|multiarch) # group "arch", architectures 
-        (?<buildtype>release|debug)              # group"buildtype", build types +        (?<buildtype>release|debug)                     # group "buildtype", build types 
-        (?<linking>shared|static|mixed)          # group "linking", library linking/build types+        (?<linking>shared|static|mixed)                 # group "linking", library linking/build types
     )     )
     |     |
Line 34: Line 34:
             (             (
                 ((?<=linux_) # if there is "linux_" before here                 ((?<=linux_) # if there is "linux_" before here
-                    ( # then match +                    # then match 
-                        debian-8 +                    (deb|rpm)_ 
-                    )_+                    (shared|static)_ 
 +                    ( 
 +                        ((?<=deb_shared_) # if there is "deb_shared_" before here 
 +                            # then match 
 +                            (jessie|stretch|buster|sid|xenial|yakkety|zesty|artful)_ 
 +                        ) 
 +                        | 
 +                        ((?<!deb_shared_) # if there is no "deb_shared_", don't match anything 
 +                        ) 
 +                    )
                 )                 )
                 |                 |
-                ((?<!linux_)) # if there is no "linux_", don't match anything+                ((?<!linux_) # if there is no "linux_", don't match anything 
 +                )
             )             )
         \g<arch>_(stable|nightly)_\g<buildtype>         \g<arch>_(stable|nightly)_\g<buildtype>
Line 46: Line 56:
 </code> </code>
  
-You can test your job name against the regex using [[https://regex101.com/r/zJ9zD1/19|this regex101 save]].+You can test your job name against the regex using [[https://regex101.com/r/zJ9zD1/32|this regex101 save]].
  
 ===== Examples of valid job names ===== ===== Examples of valid job names =====
  
 +  * ''antox_build_android_armhf_release''
 +  * ''libtoxcore_analyze_cppcheck''
   * ''libtoxcore_analyze_scan-build''   * ''libtoxcore_analyze_scan-build''
   * ''libtoxcore_build_linux_arm64_mixed_debug''   * ''libtoxcore_build_linux_arm64_mixed_debug''
-  * ''libtoxcore_build_linux_x86_64_mixed_debug'' 
   * ''libtoxcore_build_windows_x86_shared_release''   * ''libtoxcore_build_windows_x86_shared_release''
   * ''libtoxcore_src''   * ''libtoxcore_src''
-  * ''qTox_analyze_cppcheck'' +  * ''qtox_build_osx_x86-64_debug'' 
-  * ''qtox_pkg_linux_debian-8_x86-64_stable_release'' +  * ''qtox_pkg_linux_deb_shared_jessie_x86-64_stable_release'' 
-  * ''qtox_pkg_windows_x86-64_stable_release'' +  * ''qtox_pkg_linux_deb_shared_yakkety_x86-64_stable_release'' 
-  * ''utox_build_windows_x86_release'' +  * ''qtox_pkg_linux_deb_static_armel_stable_debug'' 
-  * ''utox_pkg_linux-debian-8_arm_stable_debug'' +  * ''utox_build_windows_x86-64_release'' 
-  * ''utox_pkg_osx_x86-64_nightly_debug'' +  * ''utox_pkg_linux_rpm_shared_x86-64_nightly_release'' 
-  * ''uTox_src''+  * ''utox_pkg_linux_rpm_static_x86-64_nightly_release'' 
 +  * ''utox_pkg_osx_armhf_nightly_debug'' 
 +  * ''utox_src''
  
 Note that the name part of the job (''\g<name>'') can contain lower case letters, as well as upper case ones. The general rule of thumb is to use lowercase letters for everything unless you have a good reason to not to. The reason for that rule of thumb is that most jobs initially were named in lowercase only (i.e. ''libqt5'' instead of ''libQt5'', ''libffmpeg'', instead of ''libFFmpeg'', etc.), and sometime later this was enforced by adding lowercase restriction to the regex above, but qTox and uTox maintainers wanted to have "T" letter in their jobs names, so the lowercase-only restriction in regex was lifted because of that, which makes them an exception to the general rule. Note that the name part of the job (''\g<name>'') can contain lower case letters, as well as upper case ones. The general rule of thumb is to use lowercase letters for everything unless you have a good reason to not to. The reason for that rule of thumb is that most jobs initially were named in lowercase only (i.e. ''libqt5'' instead of ''libQt5'', ''libffmpeg'', instead of ''libFFmpeg'', etc.), and sometime later this was enforced by adding lowercase restriction to the regex above, but qTox and uTox maintainers wanted to have "T" letter in their jobs names, so the lowercase-only restriction in regex was lifted because of that, which makes them an exception to the general rule.
Line 72: Line 85:
  
 Although Java's regex engine doesn't support backreference constructs ''\g<name>'', you can see us using them in the regex above. We find that construct to be quite useful, since it allows us to change a frequently repeating pattern once and it will change throughout the regex, so we wrote a [[https://jsfiddle.net/fxcjuL39/1/|pre-processor program]] that translates such multi-line, indented and non-standard commented regular expressions with backreference constructs to a single-line, no indentation, no comments and with backreference constructs replaced by corresponding named capturing group, so that the final regex could be used directly in Jenkins. Although Java's regex engine doesn't support backreference constructs ''\g<name>'', you can see us using them in the regex above. We find that construct to be quite useful, since it allows us to change a frequently repeating pattern once and it will change throughout the regex, so we wrote a [[https://jsfiddle.net/fxcjuL39/1/|pre-processor program]] that translates such multi-line, indented and non-standard commented regular expressions with backreference constructs to a single-line, no indentation, no comments and with backreference constructs replaced by corresponding named capturing group, so that the final regex could be used directly in Jenkins.
 +
 +===== Usefulness of job naming =====
 +
 +It's obvious that such job naming, which might seem a bit too explicit to some, makes it clear right away just what a job is producing. We could have stopped on that fact, since it's quite an achievement by itself, but there is a smart application of it that you can find useful when managing big amount of jobs.
 +
 +As an example, there are quite a few ''libqt5_build_*'' jobs in Jenkins:
 +
 +  - ''libqt5_build_linux_x86-64_static_debug''
 +  - ''libqt5_build_linux_x86-64_static_release''
 +  - ''libqt5_build_linux_x86_static_debug''
 +  - ''libqt5_build_linux_x86_static_release''
 +  - ''libqt5_build_windows_x86-64_shared_debug''
 +  - ''libqt5_build_windows_x86-64_shared_release''
 +  - ''libqt5_build_windows_x86-64_static_debug''
 +  - ''libqt5_build_windows_x86-64_static_release''
 +  - ''libqt5_build_windows_x86_shared_debug''
 +  - ''libqt5_build_windows_x86_shared_release''
 +  - ''libqt5_build_windows_x86_static_debug''
 +  - ''libqt5_build_windows_x86_static_release''
 +
 +Qt has many build flags that we might need to enable/disable from time to time and those flags are almost identical for all of the builds, with just a few differences: static builds pass ''-static'' flag, shared builds pass ''-shared'' flag, release builds pass ''-release'' flag, debug builds pass ''-debug'' flag, Linux x86-64 builds pass ''-platform linux-g++-64'' flag, Linux x86 builds pass ''-platform linux-g++-32'' flag and so on, but the majority of flags is the same and if we were to edit some of the flags, it's highly likely to be one of the general flags, so it would need to be changed in all of the ''libqt5'' builds, which are... well, just too many. And if the change didn't succeed, one would need to fix the change, modifying all of the jobs again. It's a pretty cumbersome exercise to do. However, there is a solution to this. If you look closely enough, you will notice that all of the special flags that differentiate one job from another are summarized in the job name. In essence, job name tell us which flags to apply, which we can use to our advantage. Jenkins defines ''$(JOB_NAME)'' environment variable (along with [[https://build.tox.chat/env-vars.html/|many other]]), which we can use to write a single bash script that will build Qt differently based on nothing but job's name! You can write such a script, make a separate job that produces just that script, make all of ''libqt5_build_*'' be downstream projects of that job and make them call the script, so that whenever the script changes, all of ''libqt5_build_*'' are rebuilt using the new version of the script.
 +
 +Here is an example of creating such a script in a Jenkins job using bash's [[http://tldp.org/LDP/abs/html/here-docs.html|Here Documents]], which you can use as a template for creating your own once-script-rules-them-all job:
 +
 +<code bash libqt5-build-script_src>
 +> ${JOB_NAME}.sh cat << "EOF"
 +#!/bin/bash
 +
 +MAKEFLAGS=j$(nproc)
 +export MAKEFLAGS
 +
 +mkdir prefix
 +cd prefix
 +PREFIX_DIR=$(pwd)
 +cd ..
 +
 +tar -xf libqt5_src.tar.xz
 +
 +cd qt*
 +
 +# set QT_CONFIGURE_OPTIONS based on ${JOB_NAME}, i.e. OS, architecture, and build types
 +
 +QT_CONFIGURE_OPTIONS="-prefix ${PREFIX_DIR} \
 +    -opensource -confirm-license \
 +    -pch \
 +    -nomake examples \
 +    -nomake tools \
 +    -nomake tests \
 +    -skip translations \
 +    -skip doc \
 +    -skip qtdeclarative \
 +    -skip activeqt \
 +    -skip qtwebsockets \
 +    -skip qtscript \
 +    -skip qtquick1 \
 +    -skip qtquickcontrols \
 +    -skip qtconnectivity \
 +    -skip qtenginio \
 +    -skip qtsensors \
 +    -skip qtserialport \
 +    -skip qtlocation \
 +    -skip qtgraphicaleffects \
 +    -skip qttools \
 +    -skip qtimageformats \
 +    -skip webkit \
 +    -skip webkit-examples \
 +    -skip webchannel \
 +    -skip multimedia \
 +    -no-openssl \
 +    -no-dbus \
 +    -no-icu \
 +    -no-qml-debug \
 +    -qt-libjpeg \
 +    -qt-libpng \
 +    -qt-zlib \
 +    -qt-pcre"
 +
 +if [[ ${JOB_NAME} == *"release"* ]]
 +then
 +  QT_CONFIGURE_OPTIONS="-release $QT_CONFIGURE_OPTIONS"
 +  
 +elif [[ ${JOB_NAME} == *"debug"* ]]
 +then
 +  QT_CONFIGURE_OPTIONS="-debug $QT_CONFIGURE_OPTIONS"
 +  
 +else
 +  echo "Error: Unknown build type, neither release or debug. Exiting."
 +  exit 1
 +fi
 +
 +
 +if [[ ${JOB_NAME} == *"static"* ]]
 +then
 +  QT_CONFIGURE_OPTIONS="-static $QT_CONFIGURE_OPTIONS"
 +  
 +elif [[ ${JOB_NAME} == *"shared"* ]]
 +then
 +  QT_CONFIGURE_OPTIONS="-shared $QT_CONFIGURE_OPTIONS"
 +  
 +else
 +  echo "Error: Unknown build type, neither static or shared. Exiting."
 +  exit 1
 +fi
 +
 +
 +if [[ ${JOB_NAME} == *"windows"* ]]
 +then
 +
 +  if [[ ${JOB_NAME} == *"x86-64"* ]]
 +  then
 +    QT_CONFIGURE_OPTIONS="-device-option CROSS_COMPILE=x86_64-w64-mingw32- $QT_CONFIGURE_OPTIONS"
 +    
 +  elif [[ ${JOB_NAME} == *"x86"* ]]
 +  then
 +    QT_CONFIGURE_OPTIONS="-device-option CROSS_COMPILE=i686-w64-mingw32- $QT_CONFIGURE_OPTIONS"
 +    
 +  else
 +    echo "Error: Support only x86 and x86-64 architectires on Windows. Exiting."
 +    exit 1
 +  fi
 +  
 +  QT_CONFIGURE_OPTIONS="-xplatform win32-g++ $QT_CONFIGURE_OPTIONS"
 +  
 +elif [[ ${JOB_NAME} == *"linux"* ]]
 +then
 +   
 +  if [[ ${JOB_NAME} == *"x86-64"* ]]
 +  then
 +    QT_CONFIGURE_OPTIONS="-platform linux-g++-64 $QT_CONFIGURE_OPTIONS"
 +    
 +  elif [[ ${JOB_NAME} == *"x86"* ]]
 +  then
 +    QT_CONFIGURE_OPTIONS="-platform linux-g++-32 $QT_CONFIGURE_OPTIONS"
 +    
 +  else
 +    echo "Error: Support only x86 and x86-64 architectires on Linux. Exiting."
 +    exit 1
 +  fi
 +  
 +  QT_CONFIGURE_OPTIONS="$QT_CONFIGURE_OPTIONS -qt-freetype -qt-xcb"
 +
 +fi
 +
 +./configure ${QT_CONFIGURE_OPTIONS}
 +make
 +make install
 +
 +cd ..
 +
 +rm -rf qt*
 +rm libqt5_src.tar.xz
 +
 +cd "${PREFIX_DIR}"
 +
 +if [[ ${JOB_NAME} == *"windows"* ]]
 +then
 +  zip ${JOB_NAME}.zip -r *
 +  mv ${JOB_NAME}.zip ..
 +elif [[ ${JOB_NAME} == *"linux"* ]]
 +then
 +  tar -cJf ${JOB_NAME}.tar.xz *
 +  mv ${JOB_NAME}.tar.xz ..
 +fi
 +
 +# break the hardcoded into Qt build path
 +mv "${PREFIX_DIR}" "${PREFIX_DIR}"_
 +EOF
 +
 +chmod +x ${JOB_NAME}.sh
 +</code>
 +
 +Note that ''${JOB_NAME}'' inside and outside of Here Documents is different. Outside it evaluate to the script's job name -- "libqt5-build-script_src", when inside it's written into a file as "${JOB_NAME}", without being evaluated, i.e. it will be evaluated when someone actually calls the "libqt5-build-script_src.sh" script.
 +
 +All of the numerous ''libqt5_build_*'' jobs now have to do, is simply call ''./libqt5-build-script_src.sh'' and archive the artifact. Also, whenever you modify the script, all of ''libqt5_build_*'' are rebuilt with the new version of the script, which is exactly what we want (if not, it's still easy enough to manually cancel the job, a lot easier than manually scheduling them to build).

Trace:

Print/export