projark deposit¶
Archives a local project folder into HPC-DME collection paths under:
/CCBR_Archive/GRIDFTP/Project_CCBR-<projectnumber>/<datatype>
Syntax¶
projark deposit \
-f /data/CCBR/projects/CCBR-12345 \
-p CCBR-12345 \
-d Analysis
Stage on a larger filesystem instead of /scratch/$USER:
projark deposit \
-f /data/CCBR/projects/CCBR-12345 \
-p CCBR-12345 \
-d Analysis \
--staging-dir /data/$USER/staging
# or set it once for the session
export PROJARK_STAGING_DIR=/data/$USER/staging
Inputs¶
-f,--folder(required): local folder to archive.-p,--projectnumber,--project-number(required): project identifier.-d,--datatype(optional):Analysis(default) orRawdata(case-insensitive).-t,--tarname(optional): override tar filename (defaultccbr<projectnumber>.tar).-s,--split-size-gb(optional): split threshold/chunk size, default500.--staging-dir(optional): base folder for staging (default$PROJARK_STAGING_DIR, else/scratch/$USER). Files are written to<staging-dir>/CCBR-<projectnumber>/<datatype>/. A custom folder must already exist and be writable.--cleanup/-k,--no-cleanup: delete the staging folder after a successful transfer; cleanup is enabled by default.
--projectnumber normalization:
- Accepts any non-empty value.
- Repeated leading
ccbrprefixes are removed (case-insensitive; each may be followed by_,-, or nothing). - Examples:
CCBR-1234->1234CCBR-abcd->abcdccbr_ccbr-1234abc->1234abc
Runtime Behavior¶
- Sync gate (
checkapisync) - Helix host check; print active
hpcdme.propertieslines for verification; abort if proxy settings are active tmux/screen/Open OnDemand graphical session check- Staging preflight, local only and before anything is created in HPC-DME (see Staging Preflight)
- Ensure target collections exist (create as needed with all required HPC-DME metadata)
- Recreate the staging folder, write the
.projark_stagingmarker, and stage tar + filelist in<base>/CCBR-<projectnumber>/<datatype>/ - Split tar if above split threshold
- Generate
.md5for staged files - Check staged files against destination vault; rename with counter if conflict found
- Transfer the
<datatype>/folder viadm_register_directory(hidden/dot files automatically excluded) - Cleanup the staging folder (default on; only if it carries the
.projark_stagingmarker)
Staging Preflight¶
Before any HPC-DME collection is created, deposit:
- resolves the staging base (
--staging-dir>$PROJARK_STAGING_DIR>/scratch/$USER); - checks that
--tarnameis a plain file name; - aborts if the staging folder
<base>/CCBR-<projectnumber>is the source folder, is inside it, or contains it; - estimates the uncompressed tar size from the source tree and aborts if free space is insufficient (see Free-Space Check);
- aborts if
<base>/CCBR-<projectnumber>already exists without a.projark_stagingmarker (see Staging Folder Marker).
Staging Folder Marker¶
deposit writes <base>/CCBR-<projectnumber>/.projark_staging (JSON with created_at, user, host, parkit_version, projectnumber, datatype, source). The marker sits outside the uploaded <datatype>/ folder, so it is never sent to HPC-DME.
- An existing
CCBR-<projectnumber>folder is deleted and recreated only if it has the marker, including under/scratch/$USER. - An existing folder without the marker makes
depositabort without touching it. Remove or rename it (after checking its contents), or choose another--staging-dir. - Cleanup also requires the marker.
- Folders left by older
parkitversions (e.g. after-k) have no marker and must be removed manually.
Free-Space Check¶
deposit estimates the uncompressed tar size from the source tree (file sizes plus tar headers and padding) and requires, on the staging filesystem:
- about 2x the tar size if the tar will exceed
--split-size-gb, becausesplitwrites the parts while the full tar still exists; - otherwise 1x the tar size;
- plus the estimated
.filelistsize (one line per archived entry; can be large for many small files); - plus a margin of 1 GiB or 1% of the tar size, whichever is larger.
Space used by an existing marked staging folder from a previous run (e.g. after -k or a failed deposit) counts as available, because it is deleted before staging starts.
The check reads filesystem free space and cannot see per-user or per-group quotas (e.g. /data/$USER, /data/CCBR). Check your quota with checkquota on Biowulf/Helix before staging on a quota-limited area.
Notes¶
- Run from
tmux,screen, or an Open OnDemand graphical session. - Disclaimer: Open OnDemand is currently available only on Biowulf compute nodes, not directly on Helix. Since
projarkis Helix-only today, usetmux/screenon Helix; Open OnDemand support is future-facing until Helix access is available. --folder FASTQand--folder FASTQ/are both valid for directories.- Relative folder paths are converted to absolute paths before use.
- For raw data, pass
--datatype rawdata. projarksends completion/failure email to$USER@nih.govfromNCICCBR@mail.nih.gov.- Proxy settings (
hpc.server.proxy.url,hpc.server.proxy.port) must be commented out inhpcdme.properties. Deposit aborts with a descriptive error if active proxy lines are found. - Conflict handling: if a staged file already exists in the destination collection, the staged files are renamed by inserting a zero-padded counter before the first extension (e.g.
ccbr1431.tar→ccbr1431_001.tar). - Split-tar conflict detection: checks for both the base tarball name and the first split-chunk suffix (
*.tar_0001) to detect prior split deposits. - Hidden files (filenames starting with
.) are automatically excluded fromdm_register_directory.