Fast AL9 rpm generation. Share the functions of uploadOBS.sh

* Change: Put all shared functions of uploadOBS.sh inside a common
    uploadUtils.sh. When a fix or an update is made, this file is to
    be copied inside the packaging/ directory of the other repositories,
    using the copyUpload script.
* New: In uploadOBS.sh, add an option to fast rebuild, on the local
    machine, a rpm. Currently the AlmaLinux 9 flavor.
      Re-created RPMs are put under <root_dir>/src/coriolis-rpm/
    Note: whenever a new "fast" version of a package is created, in
          order to be seen as "newer" than the one provides by fossEDA,
	  adjust the obsCI_CNT value to be *exactly* one above the
	  distributed version.
* New: packaging/buildDevelPackages.sh scripts to rebuild all the
    RPM packages (AL9) and push them into the 9_devel repository, ready
    for fast deployment.

documentation/content/pages/packaging/FastRpmDeployment.rst

@@ -0,0 +1,65 @@
+.. -*- Mode: rst -*-
+
+
+2. Fast Packages Deployment with RPM
+====================================
+
+This part is mostly a reminder to myself of the procedure to fast deploy
+development versions of the packages on the LIP6 internal network.
+However, the procedure could be adapted without too much effort to
+other configurations. Currently the constructed RPMs are expected to
+be for AlmaLinux 9.
+
+The script to run is: ``coriolis/packaging/buildDevelPackages.sh``.
+
+It perform the following operations:
+
+1. Rebuild the requested packages using their ``packaging/uploadOBS.sh``
+   ad-hoc scripts, with the ``--rpm`` switch.
+
+   The list of packages is stored in ``repoDirs`` and should be adjusted
+   to avoid rebuilding everything, everytime.
+
+   .. note:: **Versioning.** In order for the fast generated package to
+	     supersede the standard one provided by ``fossEDA``, increase
+	     by *exactly* one the ``obsCI_CNT`` value from the version
+	     of the base package.
+
+   .. note:: The standard RPM directory tree will be located under: ::
+
+	       <root_dir>/src/coriolis-rpm/
+	       <root_dir>/src/coriolis-rpm/SOURCES/
+	       <root_dir>/src/coriolis-rpm/BUILD/
+	       <root_dir>/src/coriolis-rpm/BUILDROOT/
+	       <root_dir>/src/coriolis-rpm/SRPMS/
+	       <root_dir>/src/coriolis-rpm/RPMS/noarch/
+	       <root_dir>/src/coriolis-rpm/RPMS/x68_64/
+
+	     The macro file defining the location is in: ::
+
+	       <root_dir>/src/coriolis-rpm/rpmmacros
+
+2. Remove the ``*debug*`` additional packages, as we don't intend to debug
+   that way.
+
+3. Sign the packages, using ``rpmsign --addsign``, the user must have a
+   ``%_gpg_name`` defined in it's ``~/.rpmmacros`` file in order for this
+   to work.
+
+4. Finaly move the package into the fast release repositories, which are,
+   in LIP6 case: ::
+
+       /dsk/l1/distributions/fossEDA/al/9_devel/Source/src
+       /dsk/l1/distributions/fossEDA/al/9_devel/x86_64/noarch
+       /dsk/l1/distributions/fossEDA/al/9_devel/x86_64/x86_64
+
+   The repositories are then rebuilt and limited to the two last versions
+   of each package with the following commands:
+
+   .. code:: bash
+
+      script> cd x86_64
+      script> createrepo_c -d --update .
+      script> dnf repomanage --keep 2 .
+  
+

documentation/content/pages/packaging/OBS.rst

@@ -5,7 +5,7 @@
 1. Packaging with openSUSE Build Service (OBS)
 ==============================================
 
-Under OBS, the build & install procedure has to be performed using ``Makefile.LIP6``.
+Under OBS, the build & install procedure has to be performed using ``Makefile.devs``.
 
 .. note:: While the git repository and the tool itself is named ``coriolis``,
 	  in order to minimize confusion, the associated package is ``coriolis-eda``.
@@ -17,7 +17,7 @@ Under OBS, the build & install procedure has to be performed using ``Makefile.LI
 
 Summary of files involved in the packaging process:
 
-* ``<>/Makefile.LIP6``
+* ``<>/Makefile.devs``
 * ``<>/packaging/coriolis-eda.spec``
 * ``<>/packaging/patchenv.sh``
 * ``<>/packaging/find_files.sh``
@@ -38,7 +38,7 @@ Summary of files involved in the packaging process:
   the build and install commands. As a consequence, once the snapshot
   is taken from a valid environment on the reference machine we only
   need to *activate* the virtual environment and run the commands right
-  away. The ``Makefile.LIP6`` has been simplificated in that way.
+  away. The ``Makefile.devs`` has been simplificated in that way.
 
 * To run |pelican| and generate the documentation, we need at least
   Python 3.8 for the ``watchfiles`` module to be available. And, as
@@ -60,7 +60,7 @@ virtual environment package removal, when needed.
 
 * As the environment is being patched, the modification time of
   the directories are made more recents. This may trick the
-  ``Makefile.LIP6`` rules to try to reinstall |pdm| or some other
+  ``Makefile.devs`` rules to try to reinstall |pdm| or some other
   dependency. In order to avoid that, we have to ``touch`` the
   relevant files. For now, |pdm|, |meson| & |pelican|.
 
@@ -78,7 +78,7 @@ event of changes in the way Coriolis is installed it must be reviewed.
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 A dedicated script has been made to handle the whole process of making
-archives and pushing them to OBS : ``mkArchives.sh``.
+archives and pushing them to OBS : ``uploadOBS.sh``.
 
 * Using the ``git-archive-all.sh`` script, an archive of the current
   ``HEAD`` of the branch we are in is taken, *along with the submodules*.

documentation/content/pages/packaging/Packaging_HTML.rst

@@ -21,4 +21,5 @@ Packaging Notes
    :depth: 1
 
 .. include:: OBS.rst
+.. include:: FastRpmDeployment.rst
 

documentation/pelicanconf.py.in

@@ -101,6 +101,7 @@ IGNORE_FILES = [ 'UsersGuide.rst'       # For User's Guide.
                , 'Name.rst'
 
                , 'OBS.rst'           # For Packaging
+               , 'FastRpmDeployment.rst'
 
                , 'December_2023.rst' # For Technical Logs
 

packaging/buildDevelPackages.sh

@@ -0,0 +1,35 @@
+#!/bin/bash
+
+ fossEDADir="/dsk/l1/distributions/fossEDA/al/9_devel"
+     rpmDir="/dsk/l1/jpc/coriolis-2.x/src/coriolis-rpm"
+
+ repoDirs=""
+ repoDirs="${repoDirs} alliance"
+#repoDirs="${repoDirs} tas-yagle"
+#repoDirs="${repoDirs} coriolis"   # Associated package is "coriolis-eda".
+#repoDirs="${repoDirs} coriolis-pdk-sky130-c4m"
+#repoDirs="${repoDirs} coriolis-pdk-ihpsg13g2"
+#repoDirs="${repoDirs} coriolis-pdk-ihpsg13g2-c4m"
+#repoDirs="${repoDirs} coriolis-pdk-gf180mcu"
+#repoDirs="${repoDirs} coriolis-pdk-gf180mcu-c4m"
+#repoDirs="${repoDirs} coriolis-pdk-nsx2"
+
+ for repoDir in ${repoDirs}; do
+   echo "Rebuilding devel packages for ${repoDir}"
+   (cd ../${repoDir}; ./packaging/uploadOBS.sh --rpm)
+ done
+
+ rm ${rpmDir}/RPMS/x86_64/*debug* 
+
+ rpmsign --addsign ${rpmDir}/SRPMS/*.rpm       \
+	           ${rpmDir}/RPMS/x86_64/*.rpm \
+                   ${rpmDir}/RPMS/noarch/*.rpm
+
+ mv ${rpmDir}/SRPMS/*.rpm       ${fossEDADir}/Source/src/
+ mv ${rpmDir}/RPMS/x86_64/*.rpm ${fossEDADir}/x86_64/x86_64/
+ mv ${rpmDir}/RPMS/noarch/*.rpm ${fossEDADir}/x86_64/noarch/
+
+ for rpmRepoDir in ${fossEDADir}/source/src \
+                   ${fossEDADir}/x86_64 ; do
+   (cd ${rpmRepoDir}; createrepo_c --update -d .; dnf repomanage --keep 2 .)
+ done

packaging/copyUpload.sh

@@ -0,0 +1,16 @@
+#!/bin/bash
+
+ repoDirs=""
+ repoDirs="${repoDirs} alliance"
+ repoDirs="${repoDirs} tas-yagle"
+ repoDirs="${repoDirs} coriolis-pdk-sky130-c4m"
+ repoDirs="${repoDirs} coriolis-pdk-ihpsg13g2"
+ repoDirs="${repoDirs} coriolis-pdk-ihpsg13g2-c4m"
+ repoDirs="${repoDirs} coriolis-pdk-gf180mcu"
+ repoDirs="${repoDirs} coriolis-pdk-gf180mcu-c4m"
+ repoDirs="${repoDirs} coriolis-pdk-nsx2"
+
+ for repoDir in ${repoDirs}; do
+   echo "./coriolis/packaging/uploadUtils.sh  -->  ../${repoDir}/packaging/"
+   cp ./packaging/uploadUtils.sh ../${repoDir}/packaging
+ done

packaging/uploadOBS.sh

@@ -1,61 +1,43 @@
 
-    version="2.5.5"
- docGithash="4e1ac7bf3"
-     obsDir="../coriolis-obs/home:jpc-lip6/coriolis-eda"
+ packageName="coriolis-eda"
+     version="2.5.5"
+  docGithash="4e1ac7bf3"
+   obsCI_CNT="248"
+    obsB_CNT="1"
 
- printHelp () {
-   echo ""
-   echo "  Usage: uploadOBSs.sh [--sources] [--docs] [--venv] [--commit] [--all]"
-   echo ""
-   echo "  Options:"
-   echo "    [--sources] : Build an archive from the HEAD of the current branch."
-   echo "    [--docs]    : Take a snapshot of the current *installed* documentation."
-   echo "                  Will be used for distributions unable to run doxygen+ghostscript"
-   echo "                  (currently openSUSE LEAP)."
-   echo "    [--venv]    : Take a snapshot of the virtual environment."
-   echo "    [--commit]  : Push the files (commit) on the remote builder repository."
-   echo "                  This will effectively triggers the rebuild of the packages."
-   echo "                  OBS local repository is hardwired to:"
-   echo "                      \"${obsDir}\""
-   echo "    [--all]     : Perform all actions at once."
-   echo ""
+ rpmSources=""
+ rpmSources="${rpmSources} packaging/coriolis-eda.spec"
+ rpmSources="${rpmSources} packaging/coriolis-eda-rpmlintrc"
+ rpmSources="${rpmSources} packaging/patchvenv.sh"
+ rpmSources="${rpmSources} packaging/find_files.sh"
+ rpmSources="${rpmSources} packaging/coloquinte-clamp.patch"
+ rpmSources="${rpmSources} packaging/coloquinte-no-eigen3.patch"
+ rpmSources="${rpmSources} packaging/lemon-no-soversion.patch"
+ rpmSources="${rpmSources} coriolis-eda-${version}.tar.gz"
+ rpmSources="${rpmSources} venv-al9-${version}.tar.gz"
+ rpmSources="${rpmSources} coriolis-docs-${docGithash}.tar.gz"
 
- }
+ debSources=""
+ debSources="${debSources} packaging/coriolis-eda.dsc"
+ debSources="${debSources} packaging/debian.copyright"
+ debSources="${debSources} packaging/debian.changelog"
+ debSources="${debSources} packaging/debian.control"
+ debSources="${debSources} packaging/debian.rules"
+ debSources="${debSources} packaging/debian.coriolis-eda.install"
+ debSources="${debSources} packaging/debian.coriolis-eda-dev.install"
+ debSources="${debSources} packaging/debian.coriolis-eda-doc.install"
 
- if [ $# -eq 0 ]; then printHelp; fi
 
-    githash=`git log -1 --pretty=format:%h`
-  doSources="false"
+ echo "Running uploadOBS.sh"
+
+ source ./packaging/uploadUtils.sh
+
      doDocs="false"
      doVEnv="false"
-   doCommit="false"
- badAgument=""
- while [ $# -gt 0 ]; do
-   case $1 in
-     --sources) doSources="true";;
-     --docs)    doDocs="true";;
-     --venv)    doVEnv="true";;
-     --commit)  doCommit="true";;
-     --all)     doSources="true"
-                doDocs="true"
-                doVEnv="true"
-                doCommit="true";;
-     *)         badArgument="$1";;
-   esac
-   shift
- done
- if [ ! -z "${badArgument}" ]; then
-   echo "[ERROR] patchenv.sh: Unknown argument \"${badArgument}\"."
-   exit 1
- fi
 
- echo "Running uploadOBSs.sh"
- echo "* Using HEAD githash as release: ${githash}."
- if [ "${doSources}" = "true" ]; then
-   echo "* Making source file archive from Git HEAD ..."
-   ./packaging/git-archive-all.sh -v --prefix coriolis-eda-${version}/ --format tar.gz coriolis-eda-${version}.tar.gz
-   #git archive --prefix=coriolis-${version}/ --format=tar.gz -o coriolis-${version}.tar.gz HEAD
- fi
+ checkRundir $0
+ parseArguments $*
+ simpleBuildArchive
  
  if [ "${doDocs}" = "true" ]; then
    echo "* Making snapshot archive of the documentation..."
@@ -68,47 +50,17 @@
  if [ "${doVEnv}" = "true" ]; then
    echo "* Making virtual environment archive from <cwd>/.venv ..."
    tar --exclude=*.pyc -zcf venv-al9-${version}.tar.gz .venv .pdm-plugins
- fi
-
-#		    packaging/debian.series                   \
-
- echo "* Update files in OBS project directory."
- echo "  OBS package directory: \"${obsDir}\"."
- for distribFile in packaging/coriolis-eda.spec               \
-                    packaging/coriolis-eda-rpmlintrc          \
-                    packaging/patchvenv.sh                    \
-                    packaging/find_files.sh                   \
-                    packaging/coloquinte-clamp.patch          \
-                    packaging/coloquinte-no-eigen3.patch      \
-                    packaging/lemon-no-soversion.patch        \
-                    coriolis-eda-${version}.tar.gz            \
-                    venv-al9-${version}.tar.gz                \
-                    coriolis-docs-${docGithash}.tar.gz        \
-		    packaging/coriolis-eda.dsc                \
-		    packaging/debian.copyright                \
-		    packaging/debian.changelog                \
-		    packaging/debian.control                  \
-		    packaging/debian.rules                    \
-		    packaging/debian.coriolis-eda.install     \
-		    packaging/debian.coriolis-eda-dev.install \
-		    packaging/debian.coriolis-eda-doc.install \
-		    ; do
-   if [ ! -f "${distribFile}" ]; then continue; fi
-   if [[ "${distribFile}" == packaging* ]]; then
-     echo "  - copy ${distribFile}."
-     cp ${distribFile} ${obsDir}
-   else
-     echo "  - move ${distribFile}."
-     mv ${distribFile} ${obsDir}
+ else
+   referenceVEnvSnapshot="../coriolis-obs/home:jpc-lip6/coriolis-eda/${venvSnapshot}"
+   if [ ! -f "${referenceVEnvSnapshot}" ]; then
+     echo "[ERROR] Venv snapshot reference not found in <${referenceVEnvSnapshot}>."
+     echo "        You must checkout the coriolis-eda project *or*, if it is already there,"
+     echo "        actually make the snapshot from it."
+     exit 1
    fi
- done
- 
- sed -i "s,^Release: *1,Release:        <CI_CNT>.<B_CNT>.${githash}," ${obsDir}/coriolis-eda.spec
- sed -i "s,^%define docGithash .*,%define docGithash ${docGithash},"  ${obsDir}/coriolis-eda.spec
- if [ "${doCommit}" = "true" ]; then
-   pushd ${obsDir}
-   osc add *
-   osc commit
-   popd
+   cp ${referenceVEnvSnapshot} .
  fi
 
+ copyFiles
+ buildObs
+ buildLocalRpm

packaging/uploadUtils.sh

@@ -0,0 +1,196 @@
+
+#
+# uploadUtils.sh provides the shared/helpers functions upon which
+# uploadOBS is built.
+#
+
+     obsDir="../coriolis-obs/home:jpc-lip6/${packageName}"
+     rpmDir="../coriolis-rpm"
+    githash=`git log -1 --pretty=format:%h`
+
+   buildRpm="false"
+   buildObs="true"
+  doSources="false"
+     doDocs="N/A"
+     doVEnv="N/A"
+   doCommit="false"
+
+ repoName="${packageName}"
+ if [ "${packageName}" = "coriolis-eda" ]; then
+   repoName="coriolis"
+ fi
+
+
+ printHelp () {
+   echo ""
+   echo "  Usage: uploadOBSs.sh [--sources] [--venv] [--commit] [--all]"
+   echo ""
+   echo "  Options:"
+   echo "    [--rpm]     : Direct build of a AL9 rpm on the local machine."
+   echo "    [--sources] : Build an archive from the HEAD of the current branch."
+   echo "    [--venv]    : Copy the venv snapshot from coriolis-eda OBS local checkout."
+   echo "                    <./coriolis-obs/home:jpc-lip6/coriolis-eda/${venvSnapshot}>"
+   if [ "${doDocs}" != "N/A" ]; then
+   echo "    [--docs]    : Take a snapshot of the current *installed* documentation."
+   echo "                  Will be used for distributions unable to run doxygen+ghostscript"
+   echo "                  (currently openSUSE LEAP)."
+   fi
+   echo "    [--commit]  : Push the files (commit) on the remote builder repository."
+   echo "                  This will effectively triggers the rebuild of the packages."
+   echo "                  OBS local repository is hardwired to:"
+   echo "                      \"${obsDir}\""
+   echo "    [--all]     : Perform all actions at once."
+   echo ""
+ }
+
+
+ setConfig () {
+   configName="$1"
+   if [ "${!configName}" = "N/A" ]; then return; fi
+   eval ${configName}="$2"
+ }
+
+
+ checkRundir ()
+ {
+   runDir=`pwd`
+   parent=`basename ${runDir}`
+   if [ "${parent}" != "${repoName}" ]; then
+     echo "[ERROR] uploadOBS.sh must be run from the root directory of the package."
+     echo "        That is, for example:"
+     echo "           ego@home:${packageName}> ./packaging/uploadOBS --rpm"
+     echo ""
+     echo "        Not from \"${runDir}\"."
+     exit 1
+   fi
+ }
+
+
+ parseArguments () {
+   if [ $# -eq 0 ]; then printHelp; exit 0; fi
+
+   badAgument=""
+   while [ $# -gt 0 ]; do
+     case $1 in
+       --help)    printHelp; exit 0;;
+       --rpm)     setConfig "buildRpm"  "true"
+                  setConfig "buildObs"  "false"
+                  setConfig "doSources" "true"
+                  setConfig "doDocs"    "true"
+                  setConfig "doVEnv"    "true"
+                  obsDir="${rpmDir}/SOURCES";;
+       --sources) setConfig "doSources" "true";;
+       --commit)  setConfig "doCommit"  "true";;
+       --all)     setConfig "doSources" "true"
+                  setConfig "doDocs"    "true"
+                  setConfig "doVEnv"    "true"
+		  setConfig "doCommit"  "true";;
+       *)         badArgument="$1";;
+     esac
+     shift
+   done
+   if [ ! -z "${badArgument}" ]; then
+     echo "[ERROR] uploadOBS.sh: Unknown argument \"${badArgument}\"."
+     exit 1
+   fi
+
+   echo "  buildRpm=\"${buildRpm}\""
+   echo "  buildObs=\"${buildObs}\""
+   echo " doSources=\"${doSources}\""
+   echo "    doDocs=\"${doDocs}\""
+   echo "    doVEnv=\"${doVEnv}\""
+   echo "  doCommit=\"${doVEnv}\""
+ }
+
+
+ runDoVEnv ()
+ {
+   if [ "${doVEnv}" = "true" ]; then
+     if [ -f "./${venvSnapshot}" ]; then
+       echo "* Venv snaphot already copied."
+     else
+       referenceVEnvSnapshot="../coriolis-obs/home:jpc-lip6/coriolis-eda/${venvSnapshot}"
+       if [ ! -f "${referenceVEnvSnapshot}" ]; then
+         echo "[ERROR] Venv snapshot reference not found in <${referenceVEnvSnapshot}>."
+         echo "        You must checkout the coriolis-eda project *or*, if it is already there,"
+         echo "        actually make the snapshot from it."
+         exit 1
+       fi
+       cp ${referenceVEnvSnapshot} .
+     fi
+   fi
+ }
+
+
+ simpleBuildArchive ()
+ {
+   echo "* Using HEAD githash as release: ${githash}."
+   if [ "${doSources}" = "true" ]; then
+     echo "* Making source file archive from Git HEAD ..."
+     ./packaging/git-archive-all.sh -v --prefix ${packageName}-${version}/ \
+                                       --format tar.gz                     \
+                                       ${packageName}-${version}.tar.gz
+   fi
+ }
+
+
+ copyFiles ()
+ {
+   buildName="RPM (local)"
+   sources="${rpmSources}"
+   if [ "${buildObs}" = "true" ]; then
+     buildName="OBS"
+     sources="${sources} ${debSources}"
+   fi
+
+   echo "* Update files in ${buildName} project directory."
+   echo "  OBS/RPM package directory: \"${obsDir}\"."
+   for distribFile in ${sources} ; do
+     if [ ! -f "${distribFile}" ]; then continue; fi
+     if [[ "${distribFile}" == packaging* ]]; then
+       echo "  - copy ${distribFile}."
+       cp ${distribFile} ${obsDir}
+     else
+       echo "  - move ${distribFile}."
+       mv ${distribFile} ${obsDir}
+     fi
+   done
+ }
+
+
+ buildObs () {
+   if [ "${buildObs}" != "true" ]; then return; fi
+
+   sed -i "s,^Release: .*,Release:        <CI_CNT>.<B_CNT>.${githash}," ${obsDir}/{packageName}.spec
+   if [ "${doDocs}" != "N/A" ]; then
+     sed -i "s,^%define docGithash .*,%define docGithash ${docGithash},"  ${obsDir}/${packageName}.spec
+   fi
+   if [ "${doCommit}" = "true" ]; then
+     pushd ${obsDir}
+     osc add *
+     osc commit
+     popd
+   fi
+ }
+
+
+ buildLocalRpm () {
+   if [ "${buildRpm}" != "true" ]; then return; fi
+
+   rpmMacros="${rpmDir}/rpmmacros"
+    specFile="${obsDir}/${packageName}.spec"
+
+   sed -i "s,^Release: .*,Release:        ${obsCI_CNT}.${obsB_CNT}.${githash},"   ${specFile}
+   if [ "${doDocs}" != "N/A" ]; then
+     sed -i "s,^%define docGithash .*,%define docGithash ${docGithash},"  ${obsDir}/${packageName}.spec
+   fi
+
+   rpmbuild --load ${rpmMacros} -ba --clean ${specFile}
+
+   for distribFile in ${rpmSources} ; do
+     rpmSourceFile="${obsDir}/`basename ${distribFile}`"   
+     if [ ! -f "${rpmSourceFile}" ]; then continue; fi
+     echo "  - remove ${rpmSourceFile}."
+     rm ${rpmSourceFile}
+   done
+ }