Docker persistent MAC address generation problem

When I was try start 150+ docker containers with docker-compose I found strange problem: "Could not generate persistent MAC address"
Could not generate persistent MAC address for veth: No such file or directory
After long googling and research I found lot of simillar issues..., same problem became in many virtualization systems like Docker, KVM... Than I found root cause of problem, too fast actions in linux network stack. So, lot of workarounds, like sleep on network subsystem action (Not help)
# cat /etc/udev/rules.d/01-net-setup-link.rules
SUBSYSTEM=="net", ACTION=="add|change", ENV{INTERFACE}=="br-*", PROGRAM="/bin/sleep 10"
SUBSYSTEM=="net", ACTION=="add|change", ENV{INTERFACE}=="docker[0-9]*", PROGRAM="/bin/sleep 10"
pre-generate MAC addresses in docker-compose.yml. (Not help and not work at all...)
services:
 test:
  image: busybox
  command: sleep 3600
  mac_address: 00:18:8B:0D:4F:0B
  networks:
  - hello
or rebuild linux modules or docker with patches... (I don't think is a good idea.)

From my point of view much better implement slow-start for docker compose service and roll back after official fix.

Before
# cat /etc/systemd/system/docker-compose.service
[Unit]
Description=Docker compose service
Requires=docker.service network-online.target
After=docker.service network-online.target

[Service]
WorkingDirectory=/srv/docker-compose

ExecStartPre=/usr/local/bin/docker-compose down -v
ExecStartPre=/usr/local/bin/docker-compose rm -fv

ExecStart=/usr/local/bin/docker-compose up

ExecStop=/usr/local/bin/docker-compose down -v

[Install]
WantedBy=multi-user.target
After (Experimental solution)
# cat /etc/systemd/system/docker-compose.service
[Unit]
Description=Docker compose slow start service
Requires=docker.service network-online.target
After=docker.service network-online.target

[Service]

PIDFile=/var/run/docker-compose.pid

WorkingDirectory=/srv/docker-compose

ExecStartPre=/usr/local/bin/docker-compose down -v
ExecStartPre=/usr/local/bin/docker-compose rm -fv

ExecStart=/bin/bash /srv/docker-compose/docker-compose-daemon.sh

ExecStop=/usr/local/bin/docker-compose down -v

[Install]
WantedBy=multi-user.target
# cat /srv/docker-compose/docker-compose-daemon.sh
#!/bin/bash

umask 022

echo $$ > /var/run/docker-compose.pid

bg() {
  cd /srv/docker-compose
  /usr/local/bin/docker-compose config --services | awk '{print "/usr/local/bin/docker-compose up --no-recreate -d "$1}' | sh | systemd-cat
}

bg &

while true; do
  echo "ping - $$ - `date`" | systemd-cat
  sleep 600;
done

Comments

Popular posts from this blog

Redis with failover replication