51 Opentraintimes incidents · november 2020 — official updates, affected components, duration and resolution details.
Due to a server problem, maps are running up to ~1h30m behind reality. We have fixed the underlying issue and will continue to monitor until the maps are up-to-date
This incident has been resolved.
Over the past weeks, we have been having a chronic issue with bots (likely AI bots) scraping content on our website at a rate far exceeding even our regular peak traffic levels. This results in occasional 404 errors when viewing pages on the site. We are working to limit this traffic and have put additional mitigations in place.
This incident has been resolved.
Planned maintenance on the Network Rail Open Data platform has overrun by 20 minutes. Maps will be showing outdated information.
This incident has been resolved.
A problem relating to database corruption for the OpenTrainTimes website caused users to receive 500 errors when viewing maps or train schedules between around 2230 on 15th February and 0915 on 16th February. The root cause was corruption on a disk partition on one of our database servers hosted on some of our older infrastructure. This server and its associated infrastructure was scheduled for renewal on to a more resilient platform before April 2026, however we are bringing this forward in the light of this outage.
We have not received any TRUST data from Network Rail since around 1712 today, nor have a number of other industry systems. We understand that there is a widespread outage. Maps will still be functioning.
We are now receiving TRUST messages, although any data produced between 1712 and 1927 has not been received. We will monitor for the next hour before closing this incident.
This incident has been resolved.
Automated monitoring has detected a total loss of data from the Network Rail Open Data platform to our servers. As a result, train movements are not updating on maps and train schedules will not show tracking in real-time.
We are now receiving real-time data, although there is a gap of several minutes between 1801 and 1807 where there is limited data. Maps will therefore be showing incorrect data until a train passes through each berth.
This incident has been resolved.
Engineers have identified high latency on our real-time feeds from the Network Rail Open Data platform. As a result, the real-time components of the website will be showing out-of-date or missing data.
This issue has now been resolved. We are awaiting a response from CACI, who manage the system on behalf of Network Rail, and we have also made an FOI request to help everyone understand the criticality of Open Data systems.
Due to a system failure, we do not currently have up-to-date real time running data. We are working to restore this in the coming hours.
This incident has been resolved.
From around 0100 on Friday 14th February, a number of IP addresses on the same network were scraping thousands of pages on our site, causing 404 errors to be returned to users. These addresses have been blocked and the website is now operating normally.
This incident has been resolved.
Since around 1300, we have been having issues with our data feeds from Network Rail. This appears to be an industry-wide problem, and we are working to find a resolution.
This incident has been resolved.
One of our servers was rebooted without warning at 0938 this morning, and services did not restart as normal. We have manually restarted services on this server, and it is currently processing through a backlog of data. Maps may be showing incorrect information for the next hour.
This incident has now been resolved.
Engineers have identified high latency on our real-time feeds from the Network Rail Open Data platform. As a result, the real-time components of the website will be showing out-of-date or missing data. This is a repeat of the incident this morning.
Latency has been stable since 1430 - we will keep monitoring until the end of the evening peak
Feeds have been stable for some hours and we are closing this incident. Our supplier is still working on a root cause analysis.
Engineers have identified problems with our feeds from the Network Rail Open Data platform. As a result, the real-time components of the website will be showing out-of-date or missing data.
The issue has been identified and a fix is being implemented.
This incident is now resolved. Maps may still show incorrect data for a few hours.
A problem with one of our servers has resulted in a lack of rail data to all subscribers. Engineers are investigating.
We have identified the problem and put in place a fix. We will continue to monitor.rf
This incident has been resolved.
From 1640 until approximately 1715, both of our map servers were not serving up new map data due to a common fault in the code running on each server. We have not seen this error before, but will put in place a software fix in a few days to ensure this does not happen again. Maps are now working again, and no data has been lost.
This incident has been resolved.
Between 2100 and 2300, a problem with one of our internal servers prevented any TD or TRUST messages from reaching other systems. This has now been resolved.
Engineers have identified high latency on our real-time feeds from the Network Rail Open Data platform. As a result, the real-time components of the website will be showing out-of-date or missing data.
Data is now being received as normal. Maps may show erroneous data for several hours for infrequently-occupied train describer berths.
Due to a problem with the Open Data platform operated by CACI on behalf of Network Rail, we are receiving no real-time data. Maps are currently not being updated. We are awaiting a response from CACI.
We have received configuration this issue is resolved, and was due to an issue upstream from our data provider, CACI
From 0108 this morning, we have not been receiving train movement messages from our supplier, CACI
This incident has been resolved
Engineers have identified a total loss of our real-time feeds from the Network Rail Open Data platform. As a result, the real-time components of the website will be showing out-of-date or missing data.
Our upstream data supplier, CACI, confirmed this is an issue and it is now resolved. Due to a large amount of data being replayed to us from yesterday, there is a significant backlog of data which we are processing. Map feeds are lagging by around 45 minutes but we anticipate they should have caught up to real-time in the next 90 minutes.
Maps are now showing real-time data.