diff --git a/documentation/content/pages/packaging/FastRpmDeployment.rst b/documentation/content/pages/packaging/FastRpmDeployment.rst new file mode 100644 index 000000000..47d745261 --- /dev/null +++ b/documentation/content/pages/packaging/FastRpmDeployment.rst @@ -0,0 +1,65 @@ +.. -*- Mode: rst -*- + + +2. Fast Packages Deployment with RPM +==================================== + +This part is mostly a reminder to myself of the procedure to fast deploy +development versions of the packages on the LIP6 internal network. +However, the procedure could be adapted without too much effort to +other configurations. Currently the constructed RPMs are expected to +be for AlmaLinux 9. + +The script to run is: ``coriolis/packaging/buildDevelPackages.sh``. + +It perform the following operations: + +1. Rebuild the requested packages using their ``packaging/uploadOBS.sh`` + ad-hoc scripts, with the ``--rpm`` switch. + + The list of packages is stored in ``repoDirs`` and should be adjusted + to avoid rebuilding everything, everytime. + + .. note:: **Versioning.** In order for the fast generated package to + supersede the standard one provided by ``fossEDA``, increase + by *exactly* one the ``obsCI_CNT`` value from the version + of the base package. + + .. note:: The standard RPM directory tree will be located under: :: + + /src/coriolis-rpm/ + /src/coriolis-rpm/SOURCES/ + /src/coriolis-rpm/BUILD/ + /src/coriolis-rpm/BUILDROOT/ + /src/coriolis-rpm/SRPMS/ + /src/coriolis-rpm/RPMS/noarch/ + /src/coriolis-rpm/RPMS/x68_64/ + + The macro file defining the location is in: :: + + /src/coriolis-rpm/rpmmacros + +2. Remove the ``*debug*`` additional packages, as we don't intend to debug + that way. + +3. Sign the packages, using ``rpmsign --addsign``, the user must have a + ``%_gpg_name`` defined in it's ``~/.rpmmacros`` file in order for this + to work. + +4. Finaly move the package into the fast release repositories, which are, + in LIP6 case: :: + + /dsk/l1/distributions/fossEDA/al/9_devel/Source/src + /dsk/l1/distributions/fossEDA/al/9_devel/x86_64/noarch + /dsk/l1/distributions/fossEDA/al/9_devel/x86_64/x86_64 + + The repositories are then rebuilt and limited to the two last versions + of each package with the following commands: + + .. code:: bash + + script> cd x86_64 + script> createrepo_c -d --update . + script> dnf repomanage --keep 2 . + + diff --git a/documentation/content/pages/packaging/OBS.rst b/documentation/content/pages/packaging/OBS.rst index 208bf3b39..fb40090ba 100644 --- a/documentation/content/pages/packaging/OBS.rst +++ b/documentation/content/pages/packaging/OBS.rst @@ -5,7 +5,7 @@ 1. Packaging with openSUSE Build Service (OBS) ============================================== -Under OBS, the build & install procedure has to be performed using ``Makefile.LIP6``. +Under OBS, the build & install procedure has to be performed using ``Makefile.devs``. .. note:: While the git repository and the tool itself is named ``coriolis``, in order to minimize confusion, the associated package is ``coriolis-eda``. @@ -17,7 +17,7 @@ Under OBS, the build & install procedure has to be performed using ``Makefile.LI Summary of files involved in the packaging process: -* ``<>/Makefile.LIP6`` +* ``<>/Makefile.devs`` * ``<>/packaging/coriolis-eda.spec`` * ``<>/packaging/patchenv.sh`` * ``<>/packaging/find_files.sh`` @@ -38,7 +38,7 @@ Summary of files involved in the packaging process: the build and install commands. As a consequence, once the snapshot is taken from a valid environment on the reference machine we only need to *activate* the virtual environment and run the commands right - away. The ``Makefile.LIP6`` has been simplificated in that way. + away. The ``Makefile.devs`` has been simplificated in that way. * To run |pelican| and generate the documentation, we need at least Python 3.8 for the ``watchfiles`` module to be available. And, as @@ -60,7 +60,7 @@ virtual environment package removal, when needed. * As the environment is being patched, the modification time of the directories are made more recents. This may trick the - ``Makefile.LIP6`` rules to try to reinstall |pdm| or some other + ``Makefile.devs`` rules to try to reinstall |pdm| or some other dependency. In order to avoid that, we have to ``touch`` the relevant files. For now, |pdm|, |meson| & |pelican|. @@ -78,7 +78,7 @@ event of changes in the way Coriolis is installed it must be reviewed. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A dedicated script has been made to handle the whole process of making -archives and pushing them to OBS : ``mkArchives.sh``. +archives and pushing them to OBS : ``uploadOBS.sh``. * Using the ``git-archive-all.sh`` script, an archive of the current ``HEAD`` of the branch we are in is taken, *along with the submodules*. diff --git a/documentation/content/pages/packaging/Packaging_HTML.rst b/documentation/content/pages/packaging/Packaging_HTML.rst index 337f28bc0..2735d083b 100644 --- a/documentation/content/pages/packaging/Packaging_HTML.rst +++ b/documentation/content/pages/packaging/Packaging_HTML.rst @@ -21,4 +21,5 @@ Packaging Notes :depth: 1 .. include:: OBS.rst +.. include:: FastRpmDeployment.rst diff --git a/documentation/pelicanconf.py.in b/documentation/pelicanconf.py.in index 7dfe2da94..5e9a2f629 100644 --- a/documentation/pelicanconf.py.in +++ b/documentation/pelicanconf.py.in @@ -101,6 +101,7 @@ IGNORE_FILES = [ 'UsersGuide.rst' # For User's Guide. , 'Name.rst' , 'OBS.rst' # For Packaging + , 'FastRpmDeployment.rst' , 'December_2023.rst' # For Technical Logs diff --git a/packaging/buildDevelPackages.sh b/packaging/buildDevelPackages.sh new file mode 100755 index 000000000..be46fb37b --- /dev/null +++ b/packaging/buildDevelPackages.sh @@ -0,0 +1,35 @@ +#!/bin/bash + + fossEDADir="/dsk/l1/distributions/fossEDA/al/9_devel" + rpmDir="/dsk/l1/jpc/coriolis-2.x/src/coriolis-rpm" + + repoDirs="" + repoDirs="${repoDirs} alliance" +#repoDirs="${repoDirs} tas-yagle" +#repoDirs="${repoDirs} coriolis" # Associated package is "coriolis-eda". +#repoDirs="${repoDirs} coriolis-pdk-sky130-c4m" +#repoDirs="${repoDirs} coriolis-pdk-ihpsg13g2" +#repoDirs="${repoDirs} coriolis-pdk-ihpsg13g2-c4m" +#repoDirs="${repoDirs} coriolis-pdk-gf180mcu" +#repoDirs="${repoDirs} coriolis-pdk-gf180mcu-c4m" +#repoDirs="${repoDirs} coriolis-pdk-nsx2" + + for repoDir in ${repoDirs}; do + echo "Rebuilding devel packages for ${repoDir}" + (cd ../${repoDir}; ./packaging/uploadOBS.sh --rpm) + done + + rm ${rpmDir}/RPMS/x86_64/*debug* + + rpmsign --addsign ${rpmDir}/SRPMS/*.rpm \ + ${rpmDir}/RPMS/x86_64/*.rpm \ + ${rpmDir}/RPMS/noarch/*.rpm + + mv ${rpmDir}/SRPMS/*.rpm ${fossEDADir}/Source/src/ + mv ${rpmDir}/RPMS/x86_64/*.rpm ${fossEDADir}/x86_64/x86_64/ + mv ${rpmDir}/RPMS/noarch/*.rpm ${fossEDADir}/x86_64/noarch/ + + for rpmRepoDir in ${fossEDADir}/source/src \ + ${fossEDADir}/x86_64 ; do + (cd ${rpmRepoDir}; createrepo_c --update -d .; dnf repomanage --keep 2 .) + done diff --git a/packaging/copyUpload.sh b/packaging/copyUpload.sh new file mode 100755 index 000000000..0da5e6cbb --- /dev/null +++ b/packaging/copyUpload.sh @@ -0,0 +1,16 @@ +#!/bin/bash + + repoDirs="" + repoDirs="${repoDirs} alliance" + repoDirs="${repoDirs} tas-yagle" + repoDirs="${repoDirs} coriolis-pdk-sky130-c4m" + repoDirs="${repoDirs} coriolis-pdk-ihpsg13g2" + repoDirs="${repoDirs} coriolis-pdk-ihpsg13g2-c4m" + repoDirs="${repoDirs} coriolis-pdk-gf180mcu" + repoDirs="${repoDirs} coriolis-pdk-gf180mcu-c4m" + repoDirs="${repoDirs} coriolis-pdk-nsx2" + + for repoDir in ${repoDirs}; do + echo "./coriolis/packaging/uploadUtils.sh --> ../${repoDir}/packaging/" + cp ./packaging/uploadUtils.sh ../${repoDir}/packaging + done diff --git a/packaging/uploadOBS.sh b/packaging/uploadOBS.sh index 9909e46ee..69a187152 100755 --- a/packaging/uploadOBS.sh +++ b/packaging/uploadOBS.sh @@ -1,61 +1,43 @@ - version="2.5.5" - docGithash="4e1ac7bf3" - obsDir="../coriolis-obs/home:jpc-lip6/coriolis-eda" + packageName="coriolis-eda" + version="2.5.5" + docGithash="4e1ac7bf3" + obsCI_CNT="248" + obsB_CNT="1" - printHelp () { - echo "" - echo " Usage: uploadOBSs.sh [--sources] [--docs] [--venv] [--commit] [--all]" - echo "" - echo " Options:" - echo " [--sources] : Build an archive from the HEAD of the current branch." - echo " [--docs] : Take a snapshot of the current *installed* documentation." - echo " Will be used for distributions unable to run doxygen+ghostscript" - echo " (currently openSUSE LEAP)." - echo " [--venv] : Take a snapshot of the virtual environment." - echo " [--commit] : Push the files (commit) on the remote builder repository." - echo " This will effectively triggers the rebuild of the packages." - echo " OBS local repository is hardwired to:" - echo " \"${obsDir}\"" - echo " [--all] : Perform all actions at once." - echo "" + rpmSources="" + rpmSources="${rpmSources} packaging/coriolis-eda.spec" + rpmSources="${rpmSources} packaging/coriolis-eda-rpmlintrc" + rpmSources="${rpmSources} packaging/patchvenv.sh" + rpmSources="${rpmSources} packaging/find_files.sh" + rpmSources="${rpmSources} packaging/coloquinte-clamp.patch" + rpmSources="${rpmSources} packaging/coloquinte-no-eigen3.patch" + rpmSources="${rpmSources} packaging/lemon-no-soversion.patch" + rpmSources="${rpmSources} coriolis-eda-${version}.tar.gz" + rpmSources="${rpmSources} venv-al9-${version}.tar.gz" + rpmSources="${rpmSources} coriolis-docs-${docGithash}.tar.gz" - } + debSources="" + debSources="${debSources} packaging/coriolis-eda.dsc" + debSources="${debSources} packaging/debian.copyright" + debSources="${debSources} packaging/debian.changelog" + debSources="${debSources} packaging/debian.control" + debSources="${debSources} packaging/debian.rules" + debSources="${debSources} packaging/debian.coriolis-eda.install" + debSources="${debSources} packaging/debian.coriolis-eda-dev.install" + debSources="${debSources} packaging/debian.coriolis-eda-doc.install" - if [ $# -eq 0 ]; then printHelp; fi - githash=`git log -1 --pretty=format:%h` - doSources="false" + echo "Running uploadOBS.sh" + + source ./packaging/uploadUtils.sh + doDocs="false" doVEnv="false" - doCommit="false" - badAgument="" - while [ $# -gt 0 ]; do - case $1 in - --sources) doSources="true";; - --docs) doDocs="true";; - --venv) doVEnv="true";; - --commit) doCommit="true";; - --all) doSources="true" - doDocs="true" - doVEnv="true" - doCommit="true";; - *) badArgument="$1";; - esac - shift - done - if [ ! -z "${badArgument}" ]; then - echo "[ERROR] patchenv.sh: Unknown argument \"${badArgument}\"." - exit 1 - fi - echo "Running uploadOBSs.sh" - echo "* Using HEAD githash as release: ${githash}." - if [ "${doSources}" = "true" ]; then - echo "* Making source file archive from Git HEAD ..." - ./packaging/git-archive-all.sh -v --prefix coriolis-eda-${version}/ --format tar.gz coriolis-eda-${version}.tar.gz - #git archive --prefix=coriolis-${version}/ --format=tar.gz -o coriolis-${version}.tar.gz HEAD - fi + checkRundir $0 + parseArguments $* + simpleBuildArchive if [ "${doDocs}" = "true" ]; then echo "* Making snapshot archive of the documentation..." @@ -68,47 +50,17 @@ if [ "${doVEnv}" = "true" ]; then echo "* Making virtual environment archive from /.venv ..." tar --exclude=*.pyc -zcf venv-al9-${version}.tar.gz .venv .pdm-plugins - fi - -# packaging/debian.series \ - - echo "* Update files in OBS project directory." - echo " OBS package directory: \"${obsDir}\"." - for distribFile in packaging/coriolis-eda.spec \ - packaging/coriolis-eda-rpmlintrc \ - packaging/patchvenv.sh \ - packaging/find_files.sh \ - packaging/coloquinte-clamp.patch \ - packaging/coloquinte-no-eigen3.patch \ - packaging/lemon-no-soversion.patch \ - coriolis-eda-${version}.tar.gz \ - venv-al9-${version}.tar.gz \ - coriolis-docs-${docGithash}.tar.gz \ - packaging/coriolis-eda.dsc \ - packaging/debian.copyright \ - packaging/debian.changelog \ - packaging/debian.control \ - packaging/debian.rules \ - packaging/debian.coriolis-eda.install \ - packaging/debian.coriolis-eda-dev.install \ - packaging/debian.coriolis-eda-doc.install \ - ; do - if [ ! -f "${distribFile}" ]; then continue; fi - if [[ "${distribFile}" == packaging* ]]; then - echo " - copy ${distribFile}." - cp ${distribFile} ${obsDir} - else - echo " - move ${distribFile}." - mv ${distribFile} ${obsDir} + else + referenceVEnvSnapshot="../coriolis-obs/home:jpc-lip6/coriolis-eda/${venvSnapshot}" + if [ ! -f "${referenceVEnvSnapshot}" ]; then + echo "[ERROR] Venv snapshot reference not found in <${referenceVEnvSnapshot}>." + echo " You must checkout the coriolis-eda project *or*, if it is already there," + echo " actually make the snapshot from it." + exit 1 fi - done - - sed -i "s,^Release: *1,Release: ..${githash}," ${obsDir}/coriolis-eda.spec - sed -i "s,^%define docGithash .*,%define docGithash ${docGithash}," ${obsDir}/coriolis-eda.spec - if [ "${doCommit}" = "true" ]; then - pushd ${obsDir} - osc add * - osc commit - popd + cp ${referenceVEnvSnapshot} . fi + copyFiles + buildObs + buildLocalRpm diff --git a/packaging/uploadUtils.sh b/packaging/uploadUtils.sh new file mode 100644 index 000000000..48f65f128 --- /dev/null +++ b/packaging/uploadUtils.sh @@ -0,0 +1,196 @@ + +# +# uploadUtils.sh provides the shared/helpers functions upon which +# uploadOBS is built. +# + + obsDir="../coriolis-obs/home:jpc-lip6/${packageName}" + rpmDir="../coriolis-rpm" + githash=`git log -1 --pretty=format:%h` + + buildRpm="false" + buildObs="true" + doSources="false" + doDocs="N/A" + doVEnv="N/A" + doCommit="false" + + repoName="${packageName}" + if [ "${packageName}" = "coriolis-eda" ]; then + repoName="coriolis" + fi + + + printHelp () { + echo "" + echo " Usage: uploadOBSs.sh [--sources] [--venv] [--commit] [--all]" + echo "" + echo " Options:" + echo " [--rpm] : Direct build of a AL9 rpm on the local machine." + echo " [--sources] : Build an archive from the HEAD of the current branch." + echo " [--venv] : Copy the venv snapshot from coriolis-eda OBS local checkout." + echo " <./coriolis-obs/home:jpc-lip6/coriolis-eda/${venvSnapshot}>" + if [ "${doDocs}" != "N/A" ]; then + echo " [--docs] : Take a snapshot of the current *installed* documentation." + echo " Will be used for distributions unable to run doxygen+ghostscript" + echo " (currently openSUSE LEAP)." + fi + echo " [--commit] : Push the files (commit) on the remote builder repository." + echo " This will effectively triggers the rebuild of the packages." + echo " OBS local repository is hardwired to:" + echo " \"${obsDir}\"" + echo " [--all] : Perform all actions at once." + echo "" + } + + + setConfig () { + configName="$1" + if [ "${!configName}" = "N/A" ]; then return; fi + eval ${configName}="$2" + } + + + checkRundir () + { + runDir=`pwd` + parent=`basename ${runDir}` + if [ "${parent}" != "${repoName}" ]; then + echo "[ERROR] uploadOBS.sh must be run from the root directory of the package." + echo " That is, for example:" + echo " ego@home:${packageName}> ./packaging/uploadOBS --rpm" + echo "" + echo " Not from \"${runDir}\"." + exit 1 + fi + } + + + parseArguments () { + if [ $# -eq 0 ]; then printHelp; exit 0; fi + + badAgument="" + while [ $# -gt 0 ]; do + case $1 in + --help) printHelp; exit 0;; + --rpm) setConfig "buildRpm" "true" + setConfig "buildObs" "false" + setConfig "doSources" "true" + setConfig "doDocs" "true" + setConfig "doVEnv" "true" + obsDir="${rpmDir}/SOURCES";; + --sources) setConfig "doSources" "true";; + --commit) setConfig "doCommit" "true";; + --all) setConfig "doSources" "true" + setConfig "doDocs" "true" + setConfig "doVEnv" "true" + setConfig "doCommit" "true";; + *) badArgument="$1";; + esac + shift + done + if [ ! -z "${badArgument}" ]; then + echo "[ERROR] uploadOBS.sh: Unknown argument \"${badArgument}\"." + exit 1 + fi + + echo " buildRpm=\"${buildRpm}\"" + echo " buildObs=\"${buildObs}\"" + echo " doSources=\"${doSources}\"" + echo " doDocs=\"${doDocs}\"" + echo " doVEnv=\"${doVEnv}\"" + echo " doCommit=\"${doVEnv}\"" + } + + + runDoVEnv () + { + if [ "${doVEnv}" = "true" ]; then + if [ -f "./${venvSnapshot}" ]; then + echo "* Venv snaphot already copied." + else + referenceVEnvSnapshot="../coriolis-obs/home:jpc-lip6/coriolis-eda/${venvSnapshot}" + if [ ! -f "${referenceVEnvSnapshot}" ]; then + echo "[ERROR] Venv snapshot reference not found in <${referenceVEnvSnapshot}>." + echo " You must checkout the coriolis-eda project *or*, if it is already there," + echo " actually make the snapshot from it." + exit 1 + fi + cp ${referenceVEnvSnapshot} . + fi + fi + } + + + simpleBuildArchive () + { + echo "* Using HEAD githash as release: ${githash}." + if [ "${doSources}" = "true" ]; then + echo "* Making source file archive from Git HEAD ..." + ./packaging/git-archive-all.sh -v --prefix ${packageName}-${version}/ \ + --format tar.gz \ + ${packageName}-${version}.tar.gz + fi + } + + + copyFiles () + { + buildName="RPM (local)" + sources="${rpmSources}" + if [ "${buildObs}" = "true" ]; then + buildName="OBS" + sources="${sources} ${debSources}" + fi + + echo "* Update files in ${buildName} project directory." + echo " OBS/RPM package directory: \"${obsDir}\"." + for distribFile in ${sources} ; do + if [ ! -f "${distribFile}" ]; then continue; fi + if [[ "${distribFile}" == packaging* ]]; then + echo " - copy ${distribFile}." + cp ${distribFile} ${obsDir} + else + echo " - move ${distribFile}." + mv ${distribFile} ${obsDir} + fi + done + } + + + buildObs () { + if [ "${buildObs}" != "true" ]; then return; fi + + sed -i "s,^Release: .*,Release: ..${githash}," ${obsDir}/{packageName}.spec + if [ "${doDocs}" != "N/A" ]; then + sed -i "s,^%define docGithash .*,%define docGithash ${docGithash}," ${obsDir}/${packageName}.spec + fi + if [ "${doCommit}" = "true" ]; then + pushd ${obsDir} + osc add * + osc commit + popd + fi + } + + + buildLocalRpm () { + if [ "${buildRpm}" != "true" ]; then return; fi + + rpmMacros="${rpmDir}/rpmmacros" + specFile="${obsDir}/${packageName}.spec" + + sed -i "s,^Release: .*,Release: ${obsCI_CNT}.${obsB_CNT}.${githash}," ${specFile} + if [ "${doDocs}" != "N/A" ]; then + sed -i "s,^%define docGithash .*,%define docGithash ${docGithash}," ${obsDir}/${packageName}.spec + fi + + rpmbuild --load ${rpmMacros} -ba --clean ${specFile} + + for distribFile in ${rpmSources} ; do + rpmSourceFile="${obsDir}/`basename ${distribFile}`" + if [ ! -f "${rpmSourceFile}" ]; then continue; fi + echo " - remove ${rpmSourceFile}." + rm ${rpmSourceFile} + done + }