alloy-prometheus-tunnel.socket stuck failed on demeter after reboot #1424

New issue

Open

opened 2025-10-18 22:18:15 +00:00 by mjm · 1 comment

mjm commented

2025-10-18 22:18:15 +00:00

Owner

I think this happens easily because hypno (the prometheus microvm) runs on demeter, so it's likely that alloy will try to connect to prometheus before the vm is started, causing the unit to fail.

I think the socket unit just needs to be more tolerant of restarting. Maybe all of the tunnel sockets do, tbh.

I think this happens easily because hypno (the prometheus microvm) runs on demeter, so it's likely that alloy will try to connect to prometheus before the vm is started, causing the unit to fail. I think the socket unit just needs to be more tolerant of restarting. Maybe all of the tunnel sockets do, tbh.

mjm commented

2025-10-18 22:45:25 +00:00

Author

Owner

Actually, looks like I already handled that in general, and this is something more subtle. At some point, the socket is ending up already bound. It's not clear what is using it.

Oct 18 21:41:33 demeter (sd-listen)[10461]: alloy-prometheus-tunnel.socket: Failed to create listening socket ([::1]:13102): Address already in use
Oct 18 21:41:33 demeter systemd[1]: alloy-prometheus-tunnel.socket: Failed to receive listening socket ([::1]:13102): Input/output error
Oct 18 21:41:33 demeter systemd[1]: alloy-prometheus-tunnel.socket: Failed to listen on sockets: Input/output error
Oct 18 21:41:33 demeter systemd[1]: alloy-prometheus-tunnel.socket: Failed with result 'resources'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ The unit alloy-prometheus-tunnel.socket has entered the 'failed' state with result 'resources'.
Oct 18 21:41:33 demeter systemd[1]: Failed to listen on Client Tunnel 'alloy-prometheus' Socket.
░░ Subject: A start job for unit alloy-prometheus-tunnel.socket has failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit alloy-prometheus-tunnel.socket has finished with a failure.
░░
░░ The job identifier is 51881 and the job result is failed.
Oct 18 21:41:33 demeter systemd[1]: Dependency failed for Client Tunnel 'alloy-prometheus'.
░░ Subject: A start job for unit alloy-prometheus-tunnel.service has failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit alloy-prometheus-tunnel.service has finished with a failure.
░░
░░ The job identifier is 51765 and the job result is dependency.
Oct 18 21:41:33 demeter systemd[1]: alloy-prometheus-tunnel.service: Job alloy-prometheus-tunnel.service/start failed with result 'dependency'.

Actually, looks like I already handled that in general, and this is something more subtle. At some point, the socket is ending up already bound. It's not clear what is using it. ``` Oct 18 21:41:33 demeter (sd-listen)[10461]: alloy-prometheus-tunnel.socket: Failed to create listening socket ([::1]:13102): Address already in use Oct 18 21:41:33 demeter systemd[1]: alloy-prometheus-tunnel.socket: Failed to receive listening socket ([::1]:13102): Input/output error Oct 18 21:41:33 demeter systemd[1]: alloy-prometheus-tunnel.socket: Failed to listen on sockets: Input/output error Oct 18 21:41:33 demeter systemd[1]: alloy-prometheus-tunnel.socket: Failed with result 'resources'. ░░ Subject: Unit failed ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ The unit alloy-prometheus-tunnel.socket has entered the 'failed' state with result 'resources'. Oct 18 21:41:33 demeter systemd[1]: Failed to listen on Client Tunnel 'alloy-prometheus' Socket. ░░ Subject: A start job for unit alloy-prometheus-tunnel.socket has failed ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ A start job for unit alloy-prometheus-tunnel.socket has finished with a failure. ░░ ░░ The job identifier is 51881 and the job result is failed. Oct 18 21:41:33 demeter systemd[1]: Dependency failed for Client Tunnel 'alloy-prometheus'. ░░ Subject: A start job for unit alloy-prometheus-tunnel.service has failed ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ A start job for unit alloy-prometheus-tunnel.service has finished with a failure. ░░ ░░ The job identifier is 51765 and the job result is dependency. Oct 18 21:41:33 demeter systemd[1]: alloy-prometheus-tunnel.service: Job alloy-prometheus-tunnel.service/start failed with result 'dependency'. ```