Docker for development: or how I learned to stop worrying and love onboarding

Onboarding new developers is time consuming and hard. Everyone has unique quirks on their machine from past software/configuration they’ve installed, removed, or overridden that makes creating consistent environment setup guides an absolute nightmare. This is how I overcame many of the issues—and you can too—by adopting Docker for development.

Adoption of Docker

While Docker has revolutionized how my team works together, its adoption hasn’t been without its share of difficulties—as my teammates will attest. However now that I know how to properly use Docker for development, onboarding new team members is an absolute breeze. The instructions for new devs goes something like this:

1. OS specific setup:

   • macOS:

     1. Install Docker Engine

   • Linux:

     1. Install Docker Engine

     2. Install Docker Compose

        Most of my projects don’t need this, see below, however it brings Docker on Linux to parity with Mac/Windows.

   • Windows (10+ Pro)

     1. Enable Hyper-V
     2. Install Docker Engine

2. Clone the repository

3. Build local dev image, run:

   docker build command listed here
   

4. Run local:

   docker run command with volume environment vars port and other goodness
   

With the repository’s readme laid out in this manner, new developers are able to follow this procedure—completely independent of assistance—and end with an entirely consistent local development environment. This builds developer confidence and allows the developer to jump right into becoming familiar with the actual code for the project.

Developers also very quickly learn that they can be completely reckless with their local development, they don’t need to fear breaking their local environment. A reset is only a docker run away.

How to organize a Docker based project poorly

When I created my first Docker project for the team I made a horrible, horrible mistake: I started the project with a docker-compose.yml file.

It seemed reasonable to me, I thought to myself “Self, there are all these Docker images on Docker Hub that give me exactly what I need. I’ll just create a docker-compose.yml file, tie these images together, add some volume mounts into the container(s) that I need to modify the code in and everything will be perfect.”

I was wrong, I was so very wrong. The worst part was that I didn’t know I was wrong because everything seemed to work. Developers were switching from the old VM development environment quickly, new devs to the project onboarded much faster than in the past, and I thought I had it all figured out.

Mistake 1: Starting with a docker-compose.yml file

Docker Compose configuration is easy to write and makes you think in a holistic way—this is a good thing, it’s also a trap.

Since I was thinking holistically, I began thinking about the containers I needed to run for my project. I started with a webserver, then a database, a reverse proxy and so-on. I then started creating directories to match up with these containers I was defining in my docker-compose.yml file that I would mount into the containers.

Example (don’t do this):

# version here (omitted)
services:
  php:
    image: php:7.2-apache
    # other container config here (omitted)
    volumes:
      - type: bind
        source: ./php/public
        target: /var/www/html
  # other services here (omitted)
# other top-level config here (omitted)

The following mistakes explain why this is the grand-daddy of all mistakes I made.

Mistake 2: Not understanding the cost of bind mounts

When mounting from the host machine into the Docker container—as I was doing in my Docker Compose file—I wasn’t just creating Docker volumes, I was creating bind mounts.

Bind mounts in Docker for Windows/Mac are terribly expensive and should be used sparingly. While everything will seem fine at first, when a certain threshold of files is passed, the lag introduced becomes extremely noticeable. This has created a whole host of attempts to solve the problem, none of which I’ve found to be tolerable—my teammates can corroborate the many hours I’ve wasted trying to fix problems we’ve encountered.

I’ve found that the best way to address this is to limit the bind mounts needed during development by using Dockerfiles to build my own dependencies for code/configuration into my own images, rather than bind mounting everything into someone-else’s images at run-time.

Note: When you have secrets, DO NOT bake them in your image, instead inject them—at run time—through environment variables.

Mistake 3: Not separating containers into separate repositories

Since my docker-compose.yml file essentially became the authority for the root of my project, the folders that I was mounting became subdirectories in the repository.

This added a lot of complexity when we introduced Continuous Delivery tooling into our workflow. Every time a push or pull-request occurred anywhere in our Git repo, all of our code sniffing, unit testing, etc had to run against the entire application stack. At first this was tolerable, however as the number of containers grew as we added additional functionality this manifested as a major problem.

Essentially what I ended up with was a monolith! It was a monolith made up of Docker containers, but it was a monolith.

Mistake 4: Not using Docker in production

When Docker was first introduced to the team, it was only used in our local environments. A build process was set up in the Ops team’s CI tool to effectively emulate our local environments to create assets to deploy to our production and production-like environments. This worked well and let the rest of the organization warm up to our team using Docker.

While not using Docker in production isn’t a mistake—and was something entirely out of my control at the time—it let me continue with bad practices that I would have otherwise encountered and had to solve earlier. Instead I exposed my teammates to bad practices for an extended period of time 😒.

How to organize a Docker based project properly

Now that I’ve gone through all many of my mistakes with Docker, I’d like to provide an outline of what I now do to overcome these issues:

1. I now start with a plan of what functionality my project needs, not with a docker-compose.yml file.

   This affords me opportunity to adequately describe my requirements instead of being tempted to jump straight into solutioning—which often gets me into trouble.

2. For each piece of functionality I need, I determine if there is an official image that does exactly what I need without any customization at all.

   If there is: then I plan to use that Docker image.

   If there isn’t:

   1. I create a Git repository for that functionality.

   2. I create a Dockerfile in that repository.

   3. I write a README.md with instructions for building and running the image in a development mode with sane bind mounts.

      The has the added benefit of making letting developers work directly on each piece of functionality and ensuring a decoupled development style/architecture.

Dockerfile examples

As time permits, I’ll add links to articles showing examples of Dockerfiles I’ve written for different applications:

• How to use Docker for Node.js development