Skip to content

D2I Admin - CSC API Pipeline - Release Process Guide

Infos for the D2I team regarding the build and release process for CSC API Pipeline project, including the needed process for prepping a new release so that anyone in D2I team can manage releases.

Introduction

CSC API Pipeline packaged and released as versioned distribution with optional Windows executables
release is automated via Git Actions, but kcik off process initiated manually by running the helper script release.sh inside a Codespace.

For consistency - version numbers managed in pyproject.toml
- git tags eg v0.2.1 drive release workflow, so running release.sh will increment release number
- git actions workflow release-and-docs.yml (auto)handles the rest

What Gets Released

Included files

Release bundle constructed in 2 ways
1. PEP 517 build system – creates Python package (.tar.gz and .whl) inside dist/ contains only package code declared in pyproject.toml and MANIFEST.in
2. Release bundle – convenience archive (release.zip) built by release.sh, containing - Python distribution files dist/*
- README.md for end‑user ref
- .env.example sample environment config
- deployment helpers PowerShell and SQL scripts used in LA setup
- optional Windows .exe built locally with PyInstaller

Excluded files

During packaging and bundling process explicitly exclude
- development caches __pycache__/, .pytest_cache/, .coverage/
- IDE files .vscode/, .idea/
- build directories build/, dist/, old *.egg-info
- test artefacts not required in distribution

Full list of include/exclude rules maintained in pyproject.toml and MANIFEST.in
Update those to change Python package contents or adjust release.sh to change wider release bundle

Release Versioning

Release follows semantic versioning (SemVer)
- MAJOR incompatible changes
- MINOR new functionality backwards compatible
- PATCH bug fixes or small updates

eg.
- 0.2.0 minor release with new features
- 0.2.1 patch release with bug fixes
- 1.0.0 stable release milestone

in practice
- tags always prefixed with v, eg v0.2.1
- version in pyproject.toml must match numeric form without v, eg 0.2.1
- release.sh keeps values in sync, updates pyproject.toml, commits if required, applies git tag

Manifest and Packaging

Python packaging toolchain via build respects two files - pyproject.toml project metadata name version dependencies - MANIFEST.in extra non‑code files to include, eg .env.example, SQL scripts

combined definition controls what end‑user receives when installing from PyPI or unzipping release file

Overview

flowchart TD A[dev runs ./release.sh] --> B[clean caches and old artefacts] B --> C[normalise version\nPEP 440 in pyproject.toml + vX.Y.Z tag] C --> D[build sdist and wheel\nartifacts in dist dir] D --> E[twine metadata check + smoke test venv] E --> F[create release_bundle and release.zip] F --> G{confirm push} G -- yes --> H[push main + tag vX.Y.Z] G -- no --> X[stop no release triggered] H --> I[git actions release-and-docs.yml] I --> J[build and test jobs] J --> K[publish artefacts and docs] K --> L[create or update git release] L --> M[release assets available for download]

Release Script release.sh

Entry point for dev/d2i, within repo root, & run from within Codespace (or Shell)

does the following

  1. confirm on main and working tree clean
  2. prompt for new version default last tag + patch bump
  3. update pyproject.toml if version changed and commit
  4. clean caches and previous build artefacts
  5. build package sdist + wheel, run metadata check, smoke test in fresh venv
  6. optionally build .exe if on Windows
  7. create release_bundle/ and release.zip
  8. prompt to tag and push git push origin main plus new tag

run via

chmod +x release.sh
./release.sh

when prompted - enter version or accept default - confirm push to origin to trigger Git Actions

git workflow release-and-docs.yml then builds and publishes release automatically

Release Steps

  • pull latest changes from main
  • run ./release.sh
  • confirm version and tag push
  • check git actions run release-and-docs.yml for success
  • validate published release artefacts

Troubleshooting

  • workflow did not trigger -> tag must start with v, eg v0.2.1
  • version mismatch -> ensure pyproject.toml matches intended tag
  • artefacts missing -> check MANIFEST.in and release.sh copy list
  • historic clutter -> old workflows and runs cleaned, only current release visible

Appendix: make release changes

  • add or remove package files -> pyproject.toml and MANIFEST.in
  • adjust bundle contents -> release.sh copy list to release_bundle/
  • change trigger logic or jobs -> .github/workflows/release-and-docs.yml
  • secret rotation and credentials -> repo or org secrets in Git settings (obv no secrets in repo!)