DNS resolution interrupted for some domains
Affected
Degraded performance from 1:00 AM to 9:25 AM, Operational from 9:25 AM to 10:45 AM
Degraded performance from 1:00 AM to 9:25 AM, Operational from 9:25 AM to 10:45 AM
- Postmortem · UTC
Summary
During the night of October 25 to 26, 2026, a file system error on the server hosting our primary nameserver (ns1.yorkdns.com) prevented it from answering queries. Our secondary nameserver (ns2.yorkdns.com) took over as intended, and the vast majority of domains remained reachable. However, some zones were not properly replicated to it:
30 zones had never been transferred to ns2. While ns1 was unavailable, these domains no longer resolved.
109 zones were served by ns2, but in a version dating from October 22: changes made since then had not been propagated.
No data was lost.
Timeline (Paris time)
Time
Event
Oct 22
Release of a new version of our DNS management tool, introducing the two replication issues described below. No visible effect while ns1 was running.
Oct 26, 02:00
Start of incident: ns1 stops responding. ns2 takes over, except for the affected zones.
07:00
First customer reports, investigation started.
10:00
Cause of the ns1 outage identified, work in progress.
10:25
ns1 repaired and back in service. All domains resolve again.
10:55
Replication issue identified. The 30 missing zones are transferred to ns2.
11:13
Fix for the 109 outdated zones validated, then applied to all of them.
11:25
All 328 zones are identical on ns1 and ns2.
11:45
Incident resolved.
Root causeThe ns1 outage revealed two issues introduced on October 22:
Wrong zone type: our DNS management tool created new zones with the “Native” type instead of “Primary”. A Native zone is never announced to the secondary server, so it never receives it.
Lowered serial number: an update operation decreased the version number (serial) of 109 zones. A secondary server only accepts an update if this number increases, so ns2 kept the old version of these zones.
Our monitoring checked the network availability of the servers, but neither their DNS answers nor the replication between them. These issues therefore remained unnoticed until the incident.
ResolutionThe ns1 server was repaired and its configuration hardened. The zones were fixed, and a zone-by-zone comparison confirms that all 328 zones answer identically on both servers.
Prevention
Tool fix: every zone is now created as a primary zone and announced to ns2 upon creation, then after every change.
Monitoring: probes that query ns1 and ns2 directly with real DNS requests, alerting us within one to two minutes if either stops responding.
We apologise to all affected customers. For any question about a specific domain, our support team remains available.
- Resolved · UTC
This incident has been resolved. All domains are served by both of our nameservers, and enhanced monitoring has been put in place. A detailed report will be published shortly.
- Update · UTC
All zones are now identical on ns1 and ns2. DNS redundancy is fully operational. We are continuing to monitor.
- Update · UTC
We identified a replication issue: some zones were not being transferred to ns2.yorkdns.com. These zones have been fixed and are now served by both servers.
- Monitoring · UTC
ns1.yorkdns.com has been repaired and is back in service. DNS resolution is restored for all domains. We are continuing to check replication between our two servers.
- Identified · UTC
Our primary nameserver ns1.yorkdns.com is experiencing a storage failure. Our secondary nameserver ns2.yorkdns.com is handling requests, but some domains are still affected. Work is in progress on ns1.
- Update · UTC
We have received reports of domains no longer resolving. Our team is investigating.
- Investigating · UTC
Some domains stopped resolving.