When you buy through affiliate links in our content, we may earn a commission at no extra cost to you. Learn 
It was meant to be a routine maintenance check – but instead, it led to the shutdown of 14 car manufacturing plants for two days straight.
Toyota’s factories went down on August 28, reducing their global output by a third, in a company famous for prolific manufacturing on tight deadlines.
The cause? Maintenance of a database gone wrong, leading the database to exceed the disk space available – both on its primary server and on the backup.
With the database out of action, Toyota could no longer make the automated requests for supplies needed to manufacture their 13,500 cars per day. And for two days, 25 production lines across the 14 factories sat empty, as the ‘glitch’ meant fresh car parts could not be ordered from their suppliers.
It might not have been a problem – until you combine that with the company’s famous “Just In Time” approach, a careful balancing act where you only receive the goods or raw materials you need just before you need them.
A Tale of “Just In Time”
Just In Time (JIT) can be an excellent ally for the right company – it maximizes inventory turnover, which in turn has many flow-on benefits to revenue and costs.
Handled with precise and surgeon-like skills, the need for warehouse space and the associated costs, such as staffing and electricity, can be a fraction of your daily or weekly output. But if the balance goes wrong in either direction, you can either end up with too many raw materials and nowhere to store them or no raw materials and machines and staff sitting idle.
The latter looks to have been the case at Toyota, where Reuters calculates a third of their global production ground to a stop. The news agency also suggests that, because of the JIT structure, it will be challenging to catch up to the two-day shortfall.
An extremely efficient production line working at close to maximum capacity every day doesn’t leave much room to boost productivity, meaning Toyota may be playing catch-up with late deliveries to customers for some time.
Seiji Sugiura, an analyst at Tokai Tokyo Research Institute, told U.S. News: “Output was running at full capacity, so there’s little additional room for production.”
Insufficient Disk Space
So, what happened on the maintenance side?
Toyota said: “The system malfunction was caused by the unavailability of some multiple servers that process parts orders.
“As for the circumstances, regular maintenance work was performed on August 27, the day before the malfunction occurred. During the maintenance procedure, data that had accumulated in the database was deleted and organized, and an error occurred due to insufficient disk space, causing the system to stop.
“Since these servers were running on the same system, a similar failure occurred in the backup function, and a switchover could not be made. This led to the suspension of domestic plant operations.
“The system was restored after the data was transferred to a server with a larger capacity on August 29, and the plants resumed operation on the following day.
“We would like to report that we have identified the above as the true cause. Countermeasures have also been put in place by replicating and verifying the situation.
“We would also like to reaffirm that the system malfunction was not caused by a cyberattack, and apologize to all parties for any concern this may have caused.”
The Bottom Line
It is a cautionary tale of regular maintenance going wrong: how a small part of the system, in this case, routine maintenance on a database, brought down the entire productivity of thousands of workers and 14 factories, with a butterfly effect of disruption to suppliers, customers, and even a temporary stock market blip.
As the world moves towards the benefits of automation and artificial intelligence (AI), it is a reminder that the more interlinked parts, the worse a domino effect can be.
Still, the system works well for them 99% of the time, so the counter-argument is an ode to efficiency during good times.
And while a company can only be as strong as its weakest component, at least this hole in the framework is patched.
Toyota added:
“Going forward, we will review our maintenance procedures and strengthen our efforts to prevent a recurrence so that we can deliver as many vehicles to our customers as soon as possible.”
