Docker Model Runner (DMR)

General

Looking at running your own LLM service on your own system to not expose your code to a third party, I came across Docker Model Runner (for short DMR).
This looked like a simple solution to run multiple LLMs on a single system.

Goal

As with all things I take on is to make it simple.
In this case a bash shell with a help script performing the commands needed to get it working.
The script should check and install the right packages for DMR, test its API and run it in the background.

Implementation

Bash Script

The script is named dmr.sh which is doing the trick. It creates a service for an LLM (Large Language Model) to run in a Docker container using a plugin package named docker-model-plugin.

Usage: dmr.sh [<options>] <command>
  Configures and runs a LLM using Docker Model Runner (DMR).

  Options:
    -h, --help : Show this help.

  Commands:
    deps        : Install dependencies.
    update      : Updates the model runner.
    pull        : Pulls the configured AI models but also starts the model runner container.
    run         : Runs the Docker model interactively on the command line.
    list        : List the AI models currently available to the model runner.
    test        : Tests the API using curl and the first AI model.
    start       : Starts running the docker model runner container in the background.
    stop        : Stops the docker model runner in the background.
    status      : Reports the status on the container.
    restart-no  : Set the container to start at host-system boot. (This is the default)
    restart-yes : Set the container to not start at host-system boot.

  Model list:
    * ai/smollm2:360M-Q4_K_M
    * ai/qwen3-coder:30B-A3B-UD-Q4_K_XL

SSL For Secure Connection

DMR provides a way to use SSL for secure connections but I think it best to just use a HTTPS proxy allowing to use acme.sh to auto generate and update certificates instead of creating self-signed certificates. acme.sh requires having an HTTP-accessible server to check the ownership of the domain name. The HTTPS port should only be accessible from the local network.

When using a VPN, the DNS server should be configured to forward the domain name to the local IP address.

TODO's

Speed Up Token Generation

A graphics card GPU speeds up LLM token generation.
So the script should check if a GPU is available and use it if it is.

Port Configuration

The script should allow configuring the ports used by DMR.

Command to Prune

The script should allow pruning unused LLM images using a prune command.

To start with, the following command lists all models available to the model runner:

docker model list --json | js .

The ones listed without a tag should be removed using docker model rm <model-id>.

Note: Why is this not part of the plugin already ?!

Conclusion

DMR is straightforward to set up and use.
It is a great tool to run multiple LLMs on a single system.