The Problem
Let’s say you have a container, say for example that it’s a container you rely on to store and sync a lot of your data. Say that one day, you notice that all of your devices stop syncing and throw time out errors. Say that you’re quite frustrated with this because to a certain extent, this container stack was supposed to be set and forget. Say that you go to the web GUI and you see this:
Why could this be? You haven’t changed anything in this stack for a few weeks and it’s been running smoothly up until now…
Let’s take a look at the logs (docker-compose logs -f):
(if you don’t have debug turned on, you’ll need it for this step - enable it and restart the container, then reload the web GUI)
The key parts to look for here are the Host field, which should contain the URL of the container that’s causing problems, and the ForwardURL field, which is where Traefik is trying to route your… traffic. I’ve whited out certain sections to protect the innocent but you can still see the key bits. Most interestingly is that when I was bringing up this container stack a few weeks ago, I noted it’s 192.168 address range. You’ll see from the screenshot that the address Traefik is trying to send to is a 172.27 address……. If you’ve ever spent any time with IP addresses, you’ll know that this is rather incorrect, and not just incorrect due to some sort of chance act or stale cache. Let’s investigate further - docker container inspect the-problem-container:
You’ll see the problem quite clearly here in the IPAddress field - Traefik is trying to route to the _backend network, which only the internals of the container stack have access to because… security. Because of that, Traefik is receiving timeouts as the interface that it has on the traefik_proxy network isn’t allowed in to the _backend network.
The Solution
Now that we’ve identified the problem, how do we fix it? We tell Traefik exactly which network it should be detecting the container on, of course! How do we do this? With a label, just like all of our other Traefik config.
- "traefik.docker.network=traefik_proxy"
Simply add this label to the problem container, recreate both containers for safety (start with the problem one, do Traefik last - and if you’ve still got debug turned on you can turn it off, it’s quite noisy otherwise) and you should be good to go. Check the web GUI and verify that you’re all back up to speed. Now go and add this label to every other container you’re using Traefik to route to just in case and you’ll set your mind at ease.
Addendum
You don’t necessarily have to add it to all of your containers, I’ve only ever seen issues with containers that have multiple networks attached. So just go and add it to the relevant ones.
SOTP: Who Got It? - ATTYA Super chill electronic music, great to work to.