Latest news

Update: Backbone monitoring and Wi-Fi redundancy

We are still observing occasional brief spikes in the core network, lasting only a few seconds, with no visible impact on traffic. We are continuing to monitor them.

Wi-Fi is currently operational, and we are still working to restore full redundancy. Until this work is complete, a disruption in our main server room could affect Wi-Fi availability.

Update: Most Wi-Fi access points back online The targeted restart was successful, and most affected Wi-Fi access points are now back online. The number still offline is only slightly above what we typically see during normal operation

Update: Wi-Fi access points currently offline Approximately 250 Wi-Fi access points are currently offline, mainly those installed in residents’ rooms. We suspect they were affected by the repeated connection drops on the Wi-Fi controller’s uplink. We are working on a way to restart only the affected access points and hope this will restore their operation. We will share an update once we know whether this resolves the issue.

Update: Backbone testing and link restoration Disabling ICMP redirects on our central switches appears to have improved backbone stability. During a test restart of heilmann-core-1, ar1-core-1 no longer became unresponsive, and the network remained stable. This is a positive result for the network, although we are continuing to monitor it. Our shared storage system (Ceph) still experienced problems during the test. One storage node was temporarily offline as expected, but many other storage components also became unavailable, leaving all virtual machines unresponsive. We are investigating why storage could not handle this interruption. We have also installed the replacement CWDM optical module on the link to Möhringen, restoring a redundant connection. Tomorrow, we will replace the module at the other end to improve the link’s stability.

We identified an additional problem with a 10 Gbit/s copper transceiver on the uplink of Wi-Fi controller AC1. It caused the connection to repeatedly drop and reconnect, resulting in brief interruptions to Selfnet Wi-Fi. This evening, we replaced the copper connection with fibre. The change caused approximately five minutes of Wi-Fi downtime. We are monitoring the connection for stability.

The Wi-Fi is working again for now; we hope to improve its stability this evening.

Unfortunately, it appears that another WLAN disruption occurred overnight. As our redundant WLAN infrastructure is currently not operational following yesterday’s relocation, the backup systems were unable to take over and mitigate the impact of the issue.

We are investigating the situation as quickly as possible and exploring options to restore WLAN service earlier than the originally planned commissioning of the backup infrastructure this evening.

Importantly, the wired LAN connections available in every room remain operational and are not affected by this incident. The disruption is currently limited to the WLAN infrastructure.

We apologize for the inconvenience and will provide further updates as soon as we have more information.

Despite the gradual improvements implemented in the network core, the Selfnet WLAN has continued to experience occasional brief disruptions, including an outage lasting a few minutes yesterday (Thursday) at approximately 18:00.

Our investigations indicate issues with the redundancy mechanisms between our WLAN controllers and gateways. Until now, these systems have been distributed across multiple geographically separate locations to provide site-level redundancy. However, this setup appears to be contributing to the intermittent disruptions under the current network conditions.

For the past six months, Selfnet has been working on a new server room located close to our primary server room in Vaihingen. The new facility was completed just a few days ago, including its uninterruptible power supply (UPS), cooling infrastructure, racks, and cabling.

Given the ongoing WLAN issues and the availability of the new facility, we decided on Thursday to relocate our WLAN backup infrastructure from our server room in Stuttgart city center to the new server room in Vaihingen. The equipment has now been physically installed at the new location, allowing us to establish redundancy through direct cabling between the two server rooms in Vaihingen.

Unfortunately, we were unable to fully commission the relocated backup systems on Thursday. We plan to complete the configuration and bring them back into operation on Friday evening.

We expect this change to improve the stability of the WLAN infrastructure and reduce the likelihood of further short disruptions. However, we will continue monitoring the situation closely to verify whether the changes have the intended effect.

The measures implemented in the network core appear to have successfully mitigated the immediate impact of the issue. As a result, the LAN has remained free of unplanned outages for approximately 36 hours.

We will continue our investigation and remediation efforts, including further work tonight, with the goal of identifying and addressing the underlying root causes rather than merely mitigating the symptoms.

While the root causes have not yet been fully resolved, the network currently appears to be operating at a satisfactory level of stability and availability outside of periods of active troubleshooting.

We will carry out further troubleshooting on the evenings of 15 and 16 October. This may cause temporary network and service interruptions. Some interruptions may be deliberately triggered as part of controlled tests to help identify the root cause. We will make every effort to keep the impact as limited and brief as possible. Thank you for your patience and understanding.

Update: Findings from 5–14 October

We are continuing to investigate the recurring outages. Our current findings point to disruptions in the backbone—the central network connecting our dormitories and servers. These disruptions also affect our shared storage system (Ceph), which several services depend on. In some cases, storage and services remain affected after the network itself has recovered.

The exact cause is still unknown. Below is a summary of what we observed and the actions taken. All times are local time in Germany.

• 5 October: A central network switch was briefly unreachable around 12:49–12:52, followed by storage communication warnings. Some Wi-Fi logins succeeded, but devices did not receive an IP address.

• 6 October: Several short network interruptions were followed by broader outages around 14:17–14:22 and 15:42–16:05, with further interruptions later in the afternoon. Multiple dormitories, servers, and web services were affected. Parts of the storage system lost contact with each other. We also recorded internal processing stalls on central switches. A separate Wi-Fi gateway problem continued into the late evening; a failed physical link is suspected there. Central switches were restarted during the evening, causing additional interruptions.

• 7 October: Another network interruption occurred around 23:13–23:19, briefly affecting Wi-Fi in Rosensteinstraße. Storage communication warnings and processing stalls on network switches occurred at the same time.

• 8 October: A broader outage around 19:40–20:00 affected the central network and several services, including DNS—the service that translates website names into network addresses. Storage warnings continued into the following day. We suspect a network disruption triggered the storage and DNS problems.

• 9 October: Around 18:37–18:46, network equipment in Rosensteinstraße, storage, DNS, login, and web services were affected. The central switch there was restarted at 18:41. Storage problems continued until approximately 18:56.

• 10 October: Two overnight outages occurred around 02:44–02:55 and 03:34–03:38, followed by repeated short interruptions in the morning. A longer incident around 15:00–17:40 affected DNS, Wi-Fi address assignment and authentication, and services including Matrix and the wiki. Our current hypothesis is that a short network disruption left the storage system unable to recover fully on its own. Manual storage restarts were required. We do not yet know why recovery stalled.

• 11 October: A network and web outage around 00:40–00:55 coincided with another switch restart in Rosensteinstraße. A further brief interruption occurred around 15:25. During the evening, we installed software updates on central switches between 21:32 and 23:03. This caused interruptions of approximately 5–30 minutes per switch, with additional effects on storage and services.

• 12 October: Work on network equipment between 16:24 and 17:12 was accompanied by network, DNS, and storage interruptions. Manual storage restarts and service recovery continued until approximately 18:00. We identified a potentially faulty uplink cable or module. After its port was disabled, fewer disturbances were reported. This is a lead, but a hardware fault has not been confirmed.

• 13 October: A planned fibre connection change affected switches at In der Au and Herriegel around 08:47–10:27. Separately, an unplanned outage around 13:54–14:01 affected Rosensteinstraße, Kade, and Theodor-Heuss-Heim. Monitoring showed dropped connections between central switches and further processing stalls. Wi-Fi authentication and login services recovered around 14:17–14:18; storage communication warnings continued until approximately 14:21.

• 14 October: At 17:53 and 17:58, Wi-Fi access points unexpectedly switched to the backup controller. Monitoring showed approximately two minutes of reduced or missing Wi-Fi traffic data during each event. Overall external traffic remained unchanged, so we cannot yet confirm whether users experienced an interruption. The reason for the controller switches is still unknown. We are considering moving the Wi-Fi systems to a newly prepared server room in the coming days, bringing the primary and backup systems closer together to improve failover reliability.

  We also found overnight peaks of approximately one million ICMP redirects per second on a central network switch. These messages tell devices to use a different network route and are handled by the switch’s CPU. A switch process was reported to use around 90% CPU. We suspect the redirects may be contributing to the recurring disruptions, but this has not been confirmed. As mitigation measures, we have disabled ICMP redirects and limited how frequently network routes can be recalculated.

  Separately, we identified a link between network switches that drops and reconnects several times a day and has a very weak optical signal. This link uses optical modules with specific CWDM wavelengths, which allow several connections to share a single fibre. Because these are uncommon wavelengths, we had no compatible replacement modules in stock. Replacement modules with higher transmit power have been ordered and are expected to arrive on Friday, 16 October. We are investigating whether this unstable link contributes to the outages.

We are investigating the recurring switch and connection problems, the high volume of ICMP redirects, and the delayed recovery of our storage system. These findings provide possible explanations, but we have not yet confirmed the root cause or whether all incidents share the same cause. We will continue monitoring the effects of the mitigation measures and share further findings here.

For about two weeks now, there have been recurring disruptions in the Selfnet backbone. We are currently troubleshooting the issue but have not yet been able to identify the cause. In most cases, the disruption results in less than one hour of downtime per day.