Docker Compute Environment

Docker logo

In this hands-on exercise, we will learn how to create and use a reproducible computational environment with Docker.

Installing Docker

You must already have Docker installed. See the Docker installation documention for more information.

In general this means having a properly configure kernel and the docker client and daemon installed on a Linux distribution. On Mac or Windows, the installation installation instructions will install a boot2docker utility to manage a VirtualBox virtual machine running the docker client and daemon.

Build the Docker Base Image

We need to build the Docker images, our reproducible computational environments. Docker images are built from a set of instructions, the Dockerfile.

After changing to the current directory scipy-tutorial-2014/environement/docker, build the base image via:

docker build -t reproducible/base .

It's also possible to use the pregenerated Docker image from the Docker Hub, via:

docker run -t -i reproducible/scipy-tutorial-2014 /bin/bash

Running Docker

To launch the base image and open a shell:

docker run -i -t reproducible/base /bin/bash

You are now in a Docker container. The -i -t flags tells docker to start up a pseudo-terminal and keep stdin open. Enter exit to exit the shell.

This runs the Docker container in the background, makes the ports listening available on the host, mounts the current working directory, and gives the container the name ipython.

Non-Interactive Containers

We can also configure containers to perform unattended tasks.

Dexy Image

We can build other images on top of reproducible/base to handle common tasks.

The Dockerfile-dexy image is used to run the dexy command.

Here's the Dockerfile:

# Runs the command "dexy"
FROM       reproducible/base
MAINTAINER Ana Nelson <ana@ananelson.com>

CMD dexy setup && dexy

It's built via:

docker build -t reproducible/dexy - < ./Dockerfile-dexy

And run via:

docker run --rm -v $PWD/../../notebooks/dexy:/home/repro reproducible/dexy

This mounts the scipy-tutorial-2014/dexy directory within the container and then runs dexy. The dexy-generated output files are present on your local file system after the container exits because of the volume mounting.

IPython Image

The Dockerfile-ipython image is used to launch the IPython notebook server.

Here's the Dockerfile:

# Runs the ipython notebook
FROM       reproducible/base
MAINTAINER Ana Nelson <ana@ananelson.com>

CMD ipython notebook --ip=* --pylab=inline --no-browser
EXPOSE 8888

It's built via:

docker build -t reproducible/ipython - < ./Dockerfile-ipython

To launch a non-interactive container which will run the IPython notebook server and forward the port IPython is running on, do:

docker run -d -P -v $PWD/../../notebooks:/home/repro --name ipython reproducible/ipython

To get information on the containers running and the ports available from a given container,

docker ps
docker port ipython 8888

If this returns 0.0.0.0:49153, we will use the 49153 port. On Linux, point your browser to http://localhost:49153. On Mac on Windows, use boot2docker to find the correct IP address,

boot2docker ip

If this returns 192.168.59.103, then point your browser to http://192.168.59.103:49153.

To stop and remove the container:

docker stop ipython
docker rm ipython

Docker Help

For a full list of docker commands or docker subcommand help, enter docker or docker subcommand with no arguments.

$ docker help
Usage: docker [OPTIONS] COMMAND [arg...]
 -H=[unix:///var/run/docker.sock]: tcp://host:port to bind/connect to or unix://path/to/socket to use

A self-sufficient runtime for linux containers.

Commands:
    attach    Attach to a running container
    build     Build an image from a Dockerfile
    commit    Create a new image from a container's changes
    cp        Copy files/folders from the containers filesystem to the host path
    diff      Inspect changes on a container's filesystem
    events    Get real time events from the server
    export    Stream the contents of a container as a tar archive
    history   Show the history of an image
    images    List images
    import    Create a new filesystem image from the contents of a tarball
    info      Display system-wide information
    inspect   Return low-level information on a container
    kill      Kill a running container
    load      Load an image from a tar archive
    login     Register or Login to the docker registry server
    logs      Fetch the logs of a container
    port      Lookup the public-facing port which is NAT-ed to PRIVATE_PORT
    pause     Pause all processes within a container
    ps        List containers
    pull      Pull an image or a repository from the docker registry server
    push      Push an image or a repository to the docker registry server
    restart   Restart a running container
    rm        Remove one or more containers
    rmi       Remove one or more images
    run       Run a command in a new container
    save      Save an image to a tar archive
    search    Search for an image in the docker index
    start     Start a stopped container
    stop      Stop a running container
    tag       Tag an image into a repository
    top       Lookup the running processes of a container
    unpause   Unpause a paused container
    version   Show the docker version information
    wait      Block until a container stops, then print its exit code

$ docker start

Usage: docker start CONTAINER [CONTAINER...]

Restart a stopped container

  -a, --attach=false         Attach container's stdout/stderr and forward all signals to the process
  -i, --interactive=false    Attach container's stdin