By Kevin Meyer
What is a day or even a week of manufacturing disruption worth to you? How about 0.07 seconds?
A split-second power disruption at a Toshiba Corp. factory in Japan could hurt shipments and raise prices for one of the world's most widely used computer chips, a mainstay of devices like smartphones, tablet PCs and digital music players.
Toshiba said the power outage could cause a 20% drop in its shipments over the next two months or so of chips known as NAND flash memory, which are used to store music, photos and data in products such as Apple Inc.'s iPhone and iPad.
I can just see the cubes filled with engineers, perhaps one of which noticed the lights flicker, going "what the…?" We've all been there, but probably not with this level of fear and trepidation. Yes they had safeguards, sort of.
Toshiba's troubles started early Wednesday when, according to power supplier Chubu Electric Power Co., there was a sudden drop in voltage that caused a 0.07-second power interruption at Toshiba's Yokkaichi memory-chip plant in Mie prefecture.
Even the briefest power interruption to the complex machines that make chips can have an effect comparable to disconnecting the power cord on a desktop computer, since the computerized controls on the systems must effectively be rebooted, said Dan Hutcheson, a chip-manufacturing analyst at VLSI Research in San Jose, Calif.
For that reason, chip companies typically take precautions that include installing what the industry calls uninterruptible power supplies. Part of Toshiba's safeguards didn't work this time because the voltage drop was more severe than what the backup system is designed to handle, a company spokesman said.
Power outages frequently cause damage to chips, which are fabricated on silicon wafers about the size of dinner plates that may take eight to 12 weeks to process, Mr. Hutcheson said. Wafers that are inside processing machines at the time of an outage are often ruined, he added, though many that are in storage or in transit among those machines are not.
That must really have been a severe 0.07 second drop. The old-fashioned analog engineer in me, who remembers the original fascination with how a tape recorder could send data to a Radio Shack TRS-80 at 60 bits per second if the volume was adjusted correctly, has a hard time fathoming how such a minute interruption can cause so much havoc. But it obviously did.
So what is that potential for an 0.07 second disruption worth? Is it even possible to prevent at any reasonable cost?
Jim Fernandez says
Several years ago when I was working as an electronic tech I remember buying and installing UPS’s (Uninterruptible power supplies). There were many different kinds.
Sounds like these guys have the wrong kind. There are UPS systems that COMPLETELY isolate your power from incoming power such that any power company problem will not affect your power. At least until you can do a slow orderly shutdown. These systems are expensive. However, it’s hard to believe that a company like this company (electronic systems manufacturing) does not have the correct kind of UPS on their machines.
Unbelievably stupid !!!
david foster says
I expect that the power consumption of this factory is fairly high, and in consequence complete UPS backup would be expensive…so either they gambled on continuity of utility power and lost, or they failed to understand up-front how sensitive their equipment was to short outages.
Speaking of lack of backup power, essentially all water pumping equipment in the US is electrically powered, and my impression is that very little of it has backup..again probably because this these are high-horsepower pumps and backup generators would be expensive. Some inherent backup is provided by gravity-fed water towers, but in a prolonged grid outage these would soon run dry.
Marc says
Ironically, Toshiba manufactures their own high power capacity double-conversion “true online” UPS systems. I’m curious to see what was installed at their factory.
The split second voltage dip is more like what you would expect to see with a line interactive or off-line UPS, which switches from the main power source to battery backup. The load from an on-line UPS always draws power from the battery through an inverter, while a rectifier connect to the mains keeps the battery charged. In this configuration voltage spikes, dips or outages have no affect on the load until the battery has discharged to its end voltage.
The on-line UPS is more expensive, so I imagine a bean counter was responsible for this decision.
Craig says
In Semiconductor manufacturing even the slightest energy disruption can be monumental. Here is a good article that explains in more detail: http://blog.vdcresearch.com/industrial_automation/2010/12/a-few-thoughts-on-the-recent-toshiba-power-outage.html.
Jim Fernandez says
Right on Marc. You would think that at least some of their most important machines would have a high power capacity double-conversion “true online” UPS system on them.
Rick says
Many several semiconductor plants have installed these for just this situation…
http://www.sandc.com/products/purewave/ups.asp
Jamie Flinchbaugh says
That is a severe problem. I’ve seen this impact in semiconductors. No one has any fun. When I was in ops, if my 120 robots lost power, they would basically forget where they were. Then you have to reset each one manually. This could take a very, very long time.
This reminds me of 2 points. First, the small things matter more than we like to think they do. We never know when one of the seemingly small things will happen to have a big impact.
Second, how do you overcome this? I think, whether you do the process or just ask the questions, FMEA (Failure Mode and Effects Analaysis) is the answer. Unless we’re going through our process with a lens towards anticipating problems, then we only find them when it’s too late. This is an example of something that has low probability but high severity.
Jim Toomey says
Looking at this from a lean standpoint, I am guessing that they had only a 20% impact because 80% of their WIP was waiting instead of working.