Skip to content

Release Notes: v3.3.0

Highlights

projark: configurable staging location with a free-space preflight

Large deposits could fill /scratch/$USER: the tarball was created, but split failed with No space left on device. Splitting writes the chunks while the full tar still exists, so peak usage is about 2× the tar size. Until this release nothing checked free space, and HPC-DME collections were created before the failure.

projark deposit -f /data/CCBR/projects/CCBR-1234 -p 1234 --staging-dir /data/$USER/staging
export PROJARK_STAGING_DIR=/data/$USER/staging   # default for deposit and retrieve
projark retrieve -p 1234 --staging-dir /data/$USER/staging
  • --staging-dir DIR (or PROJARK_STAGING_DIR) for projark deposit and projark retrieve. The default is still /scratch/$USER, and files go in <DIR>/CCBR-<projectnumber>/<datatype>/. For retrieve, --staging-dir and -f/--folder are mutually exclusive.
  • Free-space preflight: projark deposit estimates the tar and .filelist sizes and aborts before creating any HPC-DME collection if the staging filesystem lacks space. It requires about 2× the tar size when the tar will be split (else 1×), plus the .filelist and a margin. projark retrieve checks free space before downloading, including merged tars with --unsplit.
  • Safer staging:
  • deposit refuses a staging folder that is, contains, or is inside the source folder.
  • Only staging folders carrying the .projark_staging marker are ever deleted (on rerun or cleanup). A marked folder left by a previous run counts as reclaimable space.
  • retrieve never writes the marker, and refuses to download into a marked staging folder or any folder inside one.

See projark deposit, projark retrieve, Operational Guidance and Troubleshooting. (#72, #73, @kopardev)

Behavior Changes

  • An existing unmarked /scratch/$USER/CCBR-<projectnumber> folder (for example, left by an older version with -k) now makes projark deposit abort instead of being silently deleted. Check its contents and remove it manually.
  • The free-space check cannot see per-user or per-group quotas (e.g. /data/$USER, /data/CCBR). Use checkquota before large deposits.

Documentation