|
|
 |
|
|
|
register |
bbs |
search |
rss |
faq |
about
|
|
|
meet up |
add to del.icio.us |
digg it
|
|
|
|
Text about little accident at AT&T - 09/17/91
NOTICE: TO ALL CONCERNED Certain text files and messages contained on this site deal with activities and devices which would be in violation of various Federal, State, and local laws if actually carried out or constructed. The webmasters of this site do not advocate the breaking of any law. Our text files and message bases are for informational purposes only. We recommend that you contact your local law enforcement officials before undertaking any project based upon any information obtained from this or any other web site. We do not guarantee that any of the information contained on this system is correct, workable, or factual. We are not responsible for, nor do we assume any liability for, damages resulting from the use of any information on this site.
Date: Wed, 2 Oct 91 12:16:44 EDT
From: [email protected] (Michael F Eastman)
Subject: Update on 9/17/91 AT&T Outage
Organization: AT&T Bell Laboratories
The following report was posted on our internal news network by
Corporate Media Relations. It is a good summary of the events
surrounding the outage. I hope that you will find it informative.
Mike Eastman - 4ESS Development - AT&T Bell Laboratories
-----------
FOR THE RECORD *** Following is a synopsis of th e events leading
to the service disruption on Sept. 17:
Late in the afternoon on Sept. 17, the AT&T switching center
at 33 Thomas St. in lower Manhattan experienced a battery power
failure in its 20th floor power room facilities, disrupting service,
including voice and data communications for all three New York area
airports. The events leading to the disruption began earlier, between
6-7 a.m., when the Building Operations Group was contacted by Con
Edison with a request to take the facility off commercial power during
the day. We agreed to do so.
At 10:10 a.m., AT&T cut over from commercial power supplied by
Con Edison to backup, diesel-generated power. Such a cutover is
standard procedure; it is a result of the interruptible power
arrangement AT&T has with Con Edison, and was accomplished four times
without incident this summer alone, most recently on August 15 and 29.
The interruptible power arrangement with Con Edison has been in effect
formally since 1990. It capitalizes on AT&T's ability to generate at
33 Thomas St. sufficient power to cover the building's needs. By
having the means on-site to generate the building's electricity, AT&T
both protects itself from voltage brown outs that could damage
equipment and impair service, an fulfills a corporate citizenship
obligation to shed electrical load during power emergencies.
At 10:10 a.m. the AC power supervisor threw a switch, engaging
the diesel generator and taking the building off commercial power.
Throughout the building, in each of the telecommunications power
plants but one, that transfer of power from commercial AC to
diesel-generated AC, was accomplished smoothly. On the 20th floor,
where the power plant for DS3 and other high-capacity transmission
facilities is located, there was a problem. A rectifier there sensed
a spike in voltage level; to protect the power plant and facilities
the plant supported, AC power was removed from the rectifier input and
the power plant began operating on battery reserve. Subsequent tests
have determined that the overload protection relay was misadjusted
during recent plant modernization, making the shutdown circuit overly
sensitive to overvoltage. This is the only power plant in the
building that did not cutover normally.
From that moment, approximately 10:10 a.m., the batteries
supporting all DS3-and-higher-capacity facilities at 33 Thomas St.
were removed from their recharging system and were operating on
emergency reserve. That emergency reserve power is designed to last
six hours. Standard operating procedure requires the DC power
supervisor to dispatch a power technician to walk through each of the
building's power plants during a shift from commercial power to diesel
power. Had such a walkthrough occurred on Sept. 17, the technician
would have seen a "POWER" alarm in the 20th floor power room. A power
technician performs such walkthroughs as a matter of standard methods
of procedure. However, on the morning of September 17, the DC power
supervisor decided not to dispatch a technician to verify the transfer
for the following reasons:
o All six power technicians (and the supervisor) were scheduled
for a power alarm training class in another building, about
15 minutes away.
o 33 Thomas St. had not experienced a power problem in six to
eight years.
o The rectifiers had been refurbished in the last year and the
batteries were new with a six (6) hour reserve.
o Four (4) power transfers had been conducted during the summer
without problem.
Additionally, the supervisor did not arrange for a substitute by
requesting the use of one of the fifty-two power-qualified technicians
-- a technician normally charged with other duties, but capable of
responding to a power emergency -- remaining within the building.
In the absence of a power technician, if an alarm had been
recognized, one of these power-qualified technicians could have
handled the problem. Doing so would have enabled the batteries in the
20th floor power room to be recharged by the diesel generator, even as
they were being drained by providing power to the high-capacity
telecommunications facilities in the building. There was a failure to
follow standard operating procedure. Had a power technician or any
power-qualified communications technician been required to perform the
power plant walkthrough as methods of procedure mandated, the tripped
rectifier would have been discovered and reset, and a service outage
would have been avoided.
But the power plant walkthrough was not performed. All of
the building's six power plant technicians had been dispatched to
receive training, ironically, on a new computerized alarm system
that will be cut over at 33 Thomas St. in October. The equipment
for that new alarm system is functioning already at the building
where the training class was being conducted; it is being
installed, but has not yet been brought into service at 33 Thomas
St.
From 10:10 a.m. until 4:30 p.m., all high-capacity
telecommunications facilities in the building were being run on
emergency battery reserve power from the 20th floor power room. All
other equipment, such as the three 4ESS switching systems in the
building, was supplied with electricity from other power plants, and
was fully operational and functioning normally.
At 4:30 p.m., a communications technician who was just coming
on duty for the evening tour, noticed a visual display indicating the
emergency battery power condition. This visual alarm is in a location
that is normally unstaffed. At this point, the technician, who is
power qualified, made an attempt to cut back from batteries to AC
power. That attempt was unsuccessful; the batteries had been
discharged to a point where they would not physically accept
recharging current without being disconnected from the facilities they
were supporting. At 4:40 p.m., as battery life expired, those
facilities began to go down.
The restoral effort got under way virtually immediately.
During the first 30 minutes, 144 non-terminating T3 circuits, carrying
traffic passing through but not terminating in the New York area, were
restored. This amounted to some 19,200 message circuits and
approximately 1,400 private line T1 lines. By 6:00 p.m., all
equipment was disconnected from the 20th floor power plant, and
rectifiers were manually reset to force current into the batteries to
recharge them. As the rectifiers recharged the power plant,
facilities were gradually brought back on line. By 9 p.m., 43% of
domestic and 8% of international traffic was restored, by 10 p.m., 51%
of domestic and 56% of international traffic was restored, and by
midnight, virtually 100% of domestic and 95% of international traffic
was restored.
FYI:
1. The 48-volt battery plant at 33 Thomas St. is scheduled to be
replaced by the end of the year. The new plant will have restart
capability, in contrast to the existing plant.
2. A diversification of load distribution is now planned for both
call-handling systems and power systems within the local node. This
diversification will mean that any future outages would be limited to
a maximum of 50% of an office's high-capacity transmission facilities.
Rerouting is expected to be completed at 33 Thomas St. by March, 1992;
at all major metropolitan New York offices by the end of 1992, and at
all offices in the nation by the end of 1993.
3. A new power alarm system, now being installed at 33 Thomas St.,
will have built-in redundancy, with alarm connections to both the
local building and to a surveillance center in Conyers, Ga. In the
event of a failure, alarms will go off in both locations, providing a
backup if the local alarms are not functioning.
4. Nationwide, AT&T has stepped up plans to spend $200 million over
the next 12 months to improve the reliability and backup of its power
systems, which is expected to greatly diminish the risk of similar
equipment problems.
-------------
Mike Eastman att!ihlpy!mfe (708) 979-6569
AT&T Bell Laboratories Rm. 4F-328 Naperville, IL 60566
|
|
|
To the best of our knowledge, the text on this page may be freely reproduced and distributed.
If you have any questions about this, please check out our Copyright Policy.
totse.com certificate signatures
|
|
|
About | Advertise | Bad Ideas | Community | Contact Us | Copyright Policy | Drugs | Ego | Erotica
FAQ | Fringe | Link to totse.com | Search | Society | Submissions | Technology
|
|
|
|
|
|
