Using containers as development environments

CCBR Code Club

Kelly Sovacool

Oct 07, 2026

Installers & dependency managers

language environments
pip python ❌
uv, venv, pyenv python ✅
install.packages, pak R ❌
renv R ✅
conda, mamba polyglot ✅
pixi polyglot ✅

Separate environments per project for reproducibility

 

~/projects
├── CCBR-abc
├── CCBR-xyz
├── cool-python-package
└── RStatsPackage

Separate environments per project for reproducibility

 

~/projects
├── CCBR-abc
│   └── environment.yml
├── CCBR-xyz
│   ├── data-integration
│   │   └── env.yml
│   └── figures
│       └── env.yml
├── cool-python-package
│   └── pyproject.toml
└── RStatsPackage
    └── renv.lock
  • Each project specifies its own package versions.
  • Can use different environment managers for different projects.
  • Keep configuration in git so it is version-controlled & shared with others.

Containers handle dependencies at the OS level

Package managers

Install dependency versions on your local machine.

There can be slight differences in compiled versions of software depending on the hardware and operating system.

Containers

Ship with exact versions of dependencies already compiled and installed for a specific OS.

Extremely useful for pipelines; each step gets its own container. Users don’t have to worry about installing dependencies.

Containers as development environments

Architecture of Dev Containers. Source: https://code.visualstudio.com/docs/devcontainers/containers

Why use Dev Containers?

  • Dependency management for developers, including IDE extensions.
  • Consistent dependency versions across collaborators.
  • Keep genAI tools in a sandbox for security.

Why not?

  • Heavier disk usage.
  • Building takes time.
  • Not user-friendly.
  • Not intended for users, only contributors.
    • Still need to have adequate documentation of dependencies for users.

Container engines

Biowulf HPC GFE macOS
❌ ❌ Industry standard, but incompatible with HPC due to root privileges. CBIIT won’t allow Docker on macOS due to security concerns.
Sylabs Singularity ✅ ❌ Great for HPC, doesn’t support macOS. CLI not exactly identical to docker, but generally compatible.
❌ ✅ Could be installed on Biowulf in the future. CLI is a drop-in replacement for docker. Rootless by default.

How to use Dev Containers on macOS

  1. Install podman.
  2. Set up Dev Containers in VS Code.
  3. Configure Dev Containers for your projects.

Podman

Download Podman Desktop from podman.io

Podman

Install and open Podman Desktop

Podman

Create new podman machine

Podman

Increase the default resource allocation. Make sure it is rootless!

Podman

Machine is now running

Podman

Optional: auto-start the machine when podman starts

Podman

Optional: start podman when you login. Consider toggling on if you use podman daily, or toggle it off if you use it sparingly.

Podman CLI

Optional: you can configure the default machine and start it via the Podman CLI rather than the Desktop

Terminal
podman machine set --cpus 4 --disk-size 100 --memory 8192 --rootful=false
podman machine start

VS Code

Install the Dev Containers extension

VS Code

  • Open VS Code settings
  • Search for “Dev Containers” and scroll down to Docker Compose & Docker Path.
  • Set Docker Compose to podman-compose
  • Set Docker Path to full path to podman (which podman ➡️ /opt/podman/bin/podman)

Change settings to use podman instead of docker

Open a project in VS Code

File > Open or

Create a basic Dev Container configuration file

.devcontainer/devcontainer.json
{
    "image": "nciccbr/mosuite-dev:v0.5.0.9000"
}

Start the Dev Container

  • Open the VS Code command palette:
  • Select Dev Containers: Reopen in Container

Tip

Disconnect from the VPN when pulling a base container for the first time – it may interfere with the download.

Start the Dev Container

It will take some time to build the container the first time. After that it will cache the image and perform much faster.

Start the Dev Container

  • You’ll have all of the dependencies installed in the container available to use during development and testing.
  • Podman mounts the project directory to the container, and those are the only files that can be edited in the VS Code session.

Dev Container from local dockerfile

Container image from a registry:

devcontainer.json
{
    "image": "nciccbr/mosuite-dev:v0.5.0.9000"
}

Container image built from a local file:

devcontainer.json
{
  "build": {
    "dockerfile": "../docker/Dockerfile-dev",
    "context": ".."
  }
}

 

Benefit of building from a local Dockerfile: you can edit the Dockerfile and immediately use it to rebuild the Dev Container.

Using a pre-built image means if you edit the Dockerfile, you would first have to build and push a new image tag to a container registry, then rebuild the Dev Container from the new tag.

Rebuild the container when the config changes

The extension detects when the config file changes and will prompt you to rebuild the container.

Rebuild the container when the config changes

You can also trigger a rebuild via the command palette.

GenAI in Dev Containers

Rootless containers can act as a sandbox to prevent agents from accessing files in other projects.

This session had the multiOmicDataSet repo attached the workspace.

Copilot was unable to write to a file in another repo.

If it went rogue and tried to anyway, it would be incapable of doing so.

Project organization

Option 1: One Dev Container per repository, open a separate VS Code window for each repo you’re working on.

VS Code window 1
~/CCBR/mosuite
├── .devcontainer/
│   ├── devcontainer-lock.json
│   └── devcontainer.json
├── DESCRIPTION
├── docker/
│   ├── Dockerfile
│   └── Dockerfile-dev
├── R/
└── ...
VS Code window 2
~/CCBR/Tools
├── .devcontainer/
│   ├── devcontainer-lock.json
│   └── devcontainer.json
├── docker/
│   ├── Dockerfile
│   └── Dockerfile-dev
├── pyproject.toml
├── src/
└── ...

Tip

This works best when the projects are mostly unrelated and not interdependent.

Project organization

Option 2: Create a workspace with repos in subdirectories. Only need to open one VS Code window for the workspace.

VS Code workspace
~/projects/NIDAP
├── .devcontainer/
│   ├── devcontainer-lock.json
│   └── devcontainer.json
├── Code-Ocean.code-workspace
├── Code-Ocean-Capsule-Template/
├── demo-syncweaver-host-capsule/
├── demo-syncweaver-source-monorepo/
├── MOSuite-batch-correct-counts/
├── MOSuite-clean/
├── MOSuite-create/
├── MOSuite-diff-counts/
├── MOSuite-filter-counts/
├── MOSuite-filter-diff/
├── MOSuite-normalize-counts/
├── OMIX/
└── ...

Tip

This option works best for projects with multiple related and interconnected repositories. If you often need to edit code across multiple repos for a given task, set up a workspace with all of those repos.

Tip

These 2 options are not mutually-exclusive. You can mix-and-match as you see fit!

How to find examples of Dev Containers

Look for samples and template Dev Containers from established projects for inspiration.

Sample Dev Containers

Open the command palette () and search for “sample”.

https://code.visualstudio.com/docs/devcontainers/tutorial#_get-the-sample

Sample Dev Container: Python

Dev Container Features

Development container “Features” are self-contained, shareable units of installation code and Dev Container configuration. The name comes from the idea that referencing one of them allows you to quickly and easily add more tooling, runtime, or library “Features” into your development container for use by you or your collaborators.

– VS Code docs: Dev Container Features

Registry of Features: https://containers.dev/features

Dev Container Features

devcontainer.json
{
    "name": "Data Science (Python and R)",
    "image": "mcr.microsoft.com/devcontainers/base:ubuntu",
    // Features to add to the Dev Container. More info: https://containers.dev/features.
    "features": {
        "ghcr.io/devcontainers/features/python:1": {
            "version": "latest"
        },
        "ghcr.io/rocker-org/devcontainer-features/r-apt:latest": {
            "vscodeRSupport": "full",
            "installDevTools": true,
            "installRadian": true,
            "installRMarkdown": true,
            "installVscDebugger": true
        },
        "ghcr.io/rocker-org/devcontainer-features/apt-packages:1": {
            "packages": "r-cran-data.table,r-cran-tidyverse",
            "upgradePackages": true
        }
    },
    "overrideFeatureInstallOrder": [
        "ghcr.io/rocker-org/devcontainer-features/r-apt"
    ],
    // Use 'postCreateCommand' to run commands after the container is created.
    "postCreateCommand": "pip install ipykernel ipywidgets && if [ -f requirements.txt* ]; then pip install -r requirements.txt; else pip install pandas numpy matplotlib seaborn scikit-learn; fi",
    // Use 'forwardPorts' to make a list of ports inside the container available locally.
    // "forwardPorts": [],
    // Configure tool-specific properties.
    "customizations": {
        // Configure properties specific to VS Code.
        "vscode": {
            // Set *default* container specific settings.json values on container create.
            "settings": {
                "[python]": {
                    "defaultInterpreterPath": "/opt/conda/bin/python",
                    "editor.defaultFormatter": "ms-python.black-formatter",
                    "editor.formatOnType": true,
                    "editor.formatOnSave": true
                },
                "[jupyter]": {
                    "themeMatplotlibPlots": true,
                    "widgetScriptSources": [
                        "unpkg.com",
                        "jsdelivr.com"
                    ]
                },
                "[r]": {
                    "rterm.linux": "/usr/local/bin/radian",
                    "r.bracketedPaste": true
                }
            },
            // Add the IDs of extensions you want installed when the container is created.
            "extensions": [
                "ms-toolsai.jupyter",
                "ms-python.python",
                "reditorsupport.r"
            ]
        }
    },
    // Set `remoteUser` to `root` to connect as root instead. More info: https://aka.ms/vscode-remote/containers/non-root.
    "remoteUser": "vscode"
}

When you start using Features, a devcontainers-lock.json will be created alongside the config file to pin the exact Feature versions.

https://github.com/microsoft/datascience-py-r/blob/main/src/datascience-py-r/.devcontainer/devcontainer.json

What to install via Features vs in Dockerfile

  • You could install all of the dependencies via Features and not even have a Dockerfile. (See datascience-py-r on previous slide).
  • But, it is harder to control R/Python package versions via Features. You could get newer package versions when the container is rebuilt.

Dependencies of the project: Install in Dockerfile

Anything a user would need to run the code. In dependencies of pyproject.toml file, Imports of DESCRIPTION, or imported/loaded by an analysis script.

MOSuite requires DESeq2; ccbr_tools requires biopython.

IDE extensions: Install via Features

Anything unrelated to the execution of the code but assists in development and may not be universally used by all developers/contributors.

VS Code extensions, GitHub CLI, Copilot CLI, Codex CLI.

Developer dependencies: Install either way

In project.optional-dependencies of pyproject.toml or Suggests: of DESCRIPTION file. Anything that assists developers in chores: testing, linting, etc.

devtools, testthat, air for R code; pytest & ruff for Python code.

Using Git in a Dev Container

Most git commands will work, but GitHub credentials must be shared with the container in order to do git fetch/push/pull.

Terminal
$ git push

git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Important

  • If you’re using fine-grained PATs with the macOS keychain, it’ll probably work for you by default. ✅
  • If you’re using SSH keys, I haven’t figured out how to get SSH forwarding to work yet. 🚧

GitHub CLI in a Dev Container

Add the GitHub CLI Feature to your Dev Container config:

devcontainer.json
{
  "build": {
    "dockerfile": "../docker/Dockerfile-dev",
    "context": ".."
  },
  "features": {
    "ghcr.io/devcontainers/features/github-cli:1": {
      "version": "latest"
    }
  }
}

Then authenticate:

Terminal
$ gh auth login

? Where do you use GitHub? GitHub.com
? What is your preferred protocol for Git operations on this host? SSH
? Generate a new SSH key to add to your GitHub account? No
? How would you like to authenticate GitHub CLI? Login with a web browser

! Failed to copy one-time code to clipboard
  No clipboard utilities available. Please install xsel, xclip, wl-clipboard or Termux:API add-on for termux-clipboard-get/set.
! First copy your one-time code: XXXX-XXXX
Press Enter to open https://github.com/login/device in your browser...
✓ Authentication complete.
- gh config set -h github.com git_protocol ssh
✓ Configured git protocol
! Authentication credentials saved in plain text
✓ Logged in as kelly-sovacool

Now you can use gh CLI commands from inside the Dev Container.

Troubleshooting

When opening the Dev Container fails instantly, make sure you started the podman machine.

Troubleshooting

When opening the Dev Container fails instantly, make sure you started the podman machine.

Troubleshooting: insufficient resources

If processes in the Dev Container fail due to insufficient resources (memory, disk space), increase the resources for the default podman machine and restart it.

  • disk size in GB
  • memory in MB
Terminal
podman machine stop
podman machine set --cpus 4 --disk-size 112 --memory 12288 --rootful=false
podman machine start

Troubleshooting: when the build fails

When the container fails to be built, copy the log from the terminal.

Troubleshooting: use agents for help

  • Copy relevant parts of the log from the terminal and paste it into a separate window’s Copilot session.
  • Do not use the same VS Code window; you’re going to want to reload the window when trying to build the container again.

Troubleshooting: evaluate the response

Diagnosis from Copilot:

Podman’s –cache-from only accepts a repository name, while Docker also accepts a tagged image

Dev Containers in VS Code on macOS

So far I’ve shown you how to get started using Dev Containers in VS Code on macOS.

  • But what about other IDEs? RStudio, Positron
  • Other environments? Biowulf, FRCE

Positron

Positron supports Dev Containers!

…but I haven’t gotten it to work 😿

Instructions in positron’s documentation to enable development containers: https://positron.posit.co/dev-containers.html

Biowulf

Conclusion

Dev Containers have a lot of potential

  • Dev Containers can help developers make their dependency management more precise and streamlined, especially when collaborating with others.
    • There is an overhead to setting up containers, but using them is fairly smooth once built.
  • Security benefit: rootless containers can act as a sandbox for genAI agents to prevent them from reading/writing files in other projects.

Limitations prevent us from getting the most out of them right now

  • Difficulties with git authentication via SSH.
  • Lack of support/compatibility on Biowulf.
  • Difficulties with Positron.
  • GenAI is not as helpful at troubleshooting Dev Containers. Lots of red herrings.

But these could change in the future!

Resources