YorkHost - Notice history

The memory allocation issue on PVE5 has now been stabilized. Affected virtual machines have been successfully live-migrated to other nodes in our Proxmox cluster, and resource pressure on the node has returned to normal levels.

All services are operating normally. The previously announced maintenance window to install additional RAM on PVE5 will still take place as scheduled, and affected clients will be notified in advance.

Thank you for your patience.

We identified a memory allocation imbalance on our PVE5 hypervisor node, which has been impacting available RAM headroom for hosted virtual machines.

To resolve this without service disruption, we are taking the following steps:

• Live migration: Affected VMs are being live-migrated to other nodes in our Proxmox cluster. This process is transparent and does not require any reboot or downtime on your services.

• Scheduled maintenance: Once PVE5 has been safely evacuated, we will schedule a maintenance window to install additional RAM and rebalance resource allocation across the node.

No action is required on your part. We will publish the maintenance window separately and notify affected clients in advance. Services remain fully operational throughout the migration phase.

Thank you for your patience.

The GAME-NODE-07 should be back in a few minutes

SAme issue i'm looking for a temp fix.

Post-mortem — BL08 / GAME-NODE-20 (M1K02)

Summary

Node BL08, hosted in chassis M1K02 at DC2SCALE PAR3, experienced a network outage caused by hardware instability following a recent memory upgrade. Service has now been restored.

Issue 1 — Memory instability (DIMM)

Following a recent RAM addition, the server's management controller reported a critical DIMM error, which led to a system crash and an unexpected reboot. The most likely cause is a faulty memory module or a compatibility issue with the existing configuration.

Issue 2 — Network link instability (10G mezzanine)

After the memory crash, the blade's mezzanine card entered a degraded state. The ixgbe driver reported repeated link_config FAILED -22 errors at boot, and the 10G uplink to the upstream switch kept flapping between up and down, preventing stable connectivity. This behavior is consistent with a NIC left in an inconsistent state following the abrupt crash, and possibly aggravated by the same underlying memory instability.

Resolution

A full power cycle of the blade was performed, allowing the mezzanine card to reinitialize properly and the network link to come back up in a stable state. The node is now back online and serving traffic normally.

Next steps

An on-site maintenance window will be scheduled at DC2SCALE PAR3 to perform an in-depth diagnostic of the memory modules and replace any faulty DIMM if confirmed. The network interface will also be monitored closely during this period; if instability persists after the memory issue is resolved, the mezzanine card will be replaced as well.

We apologize for the inconvenience and thank you for your patience.

The issue stems from a recent RAM upgrade causing instabilities (possibly a faulty RAM module). A thorough on-site test will be performed as soon as possible. We are currently working on bringing the node back online.

The issue has been identified on our upstream provider's side at PAR5 (Digital Realty). Their teams have taken the necessary actions and connectivity has now been fully restored.

This was not an issue on our infrastructure. We will continue to monitor the situation to ensure full stability.

We apologize for the inconvenience and thank you for your patience.

We are currently experiencing a service disruption on our PAR5 (Digital Realty) site.

At this stage, we do not have a confirmed root cause. Early indications suggest it could be related to a maintenance operation on Digital Realty's side that may have encountered an issue, but this is not yet confirmed.

We are still waiting for an official update from the datacenter and our teams remain fully engaged on the matter. We will keep you posted as soon as we have more information.