The Troubled Checkout of AS-201

By late September Merritt Preston's Launch Operations staff and Rocco Petrone's Programs Division had set the 201 launch for late January. That schedule assumed the spacecraft would arrive at the Cape on 9 October. Mueller refused, at first, to approve KSC's recommended date. He hoped for an earlier launch, perhaps in late December. Although Petrone promised to continue looking for possible shortcuts, delays in the spacecraft delivery precluded a 1965 launch.12

S-IVB stage for AS-201. (1) The second stage for AS-201 arriving at pad 34, 1 October 1965.

S-IVB stage for AS-201. (2) The stage hoisted.

S-IVB stage for AS-201. (3) Eased into place.

The Douglas crew erected the S-IVB stage on 1 October and completed the initial checkout of the ground equipment test sets shortly thereafter. KSC received some bad news on the 7th: the RCA 110A computer in the breadboard at Huntsville would not be operational for another ten days. Since it would take two weeks beyond that to check out the computer's program, the 110A executive routine would not reach the Cape before 1 November. Without the executive routine to test the computer's internal systems, the 110A could not apply power to the launch vehicle. On the 15th John Twigg, chief test conductor for the Saturn IB, reported that pad operations were virtually at a standstill. Representatives from Huntsville helped devise a temporary computer program, and the launch team finally applied power to the S-IB stage on 22 October. By the end of the month, KSC had begun limited testing with an uncertified program tape. Meanwhile the instrument unit arrived and underwent inspection at hangar AF. IBM engineers corrected several deficiencies, but an environmental control system coolant pump continued to give the launch team trouble after the instrument unit was stacked above the S-IVB stage on 25 October.13

The service module for AS-201 in the operations and checkout building, November 1965.

After arrival of the major modules of spacecraft 009 in late October, the command module went to the hypergolic building for environmental control system servicing and electrical power checks. North American technicians moved the service module out to pad 16 for an electrical systems check. At the operations and checkout building, other workmen installed measuring instruments in the boilerplate lunar module adapter. Although the service module passed leak and functional tests, the 9 November static firing was postponed ten days. A dirty filter in a ground oxidizer system caused much of the delay. On the 18th Preston notified Debus that the late static firing and an accumulation of spacecraft modifications might cause a two-week slip in the launch schedule.14

The Saturn ground computer system continued to cause grief. On 10 November John Twigg reported:

The RCA 110A computer developed problems increasing in number with time of operations. It was detected that some capacitors which normally breathe in open air developed problems when under protective coating. Most of these cards [electrical printed-circuit boards] in the blockhouse computer which developed these problems first were exchanged. At the same time, it was decided to exchange identical cards in the pad computer as soon as they become available.15

We have experienced a number of high-speed memory parity errors in the last few days. No solution has been found to date. December 2nd, we have experienced some problems with random discretes to the S-IV-B stage and also apparently random outputs of the computer operating the switch selector. Investigation of this problem is under way at this time. I would like to discuss with you the feasibility to obtain the services of an outside expert (preferably some University instructor) to assess the criticality and problems of our computer system. (Army had good success with this approach, I understand.)16

With the spacecraft hot tests tying up pad 34 during daylight hours, the Saturn team continued its checkout from 7:00 p.m. to 3:00 a.m. The Saturn ground computer complex did not work any better in the dark. In five successive switch selector functional tests, the checkout team registered only a partial success. On the 15th Gruene's team traced a bad interface between the digital data acquisition system and the 110A computer to a defective printed-circuit board. After further investigations disclosed five more faulty circuit boards, Launch Vehicle Operations began a survey of all boards.** By Christmas the Saturn IB checkout had fallen 16 days behind schedule. Matters were even worse with Apollo; the spacecraft showed a 20-day slip. A January launch for AS-201 appeared highly unlikely.18

North American technicians enjoyed a one-day Christmas holiday. Sunday, the 26th, found a spacecraft team at pad 34 erecting the Apollo on top of the launch vehicle. After mounting the launch escape system, workmen linked the spacecraft with the pad facilities, checked the service module's umbilical arm fit and the white room's interface with the command module. On 5 January the spacecraft began a series of electrical tests with a launch vehicle simulator. Launch vehicle operations, during the same period, included tests of the electrical bridge wire and emergency detection system, a sequence malfunction test, and a LOX simulate and malfunction test (checking the electrical portion of the LOX system).19

A new 201 schedule, published 12 January, moved the launch date back to the week of 6 February.20 The delay was not sufficient, however, as problems continued to plague the operation. Weekly reports on computer failures ran five to six pages long. Launch officials began to wonder if they would ever get through the mission. Norman Carlson, Saturn IB Test Conductor, recalled:

It [the computer] and we had many hours of grief. You know I really predicted that we would never launch AS-201 by using the computer. It was that bad. We would power up in the morning, and sometimes we were lucky if we got two hours of testing in the whole day. It was up and down all the time.21

Computer problems with ACE and the 110A caused 13 hours of hold time during the plugs-in (umbilicals connected) test on 24 January. After similar difficulties on the plugs-out overall test, Launch Operations Manager Paul Donnelly scheduled another run for 1 February. The repeat was a success and NASA announced a launch date of 22 February. The countdown demonstration test lasted four days (3-6 February). The launch vehicle team followed a script prepared back in October:

Phase 1. Ordnance installation and S-IVB propulsion checks.

Phase 2. Battery installation, power transfer test, and guidance and control checkout.

Phase 3. Command checks and propellant loading.

Workmen hurriedly corrected a number of deficiencies found during the propellant loading and rescheduled the last 22 hours for 8-9 February. Following the flight readiness test on the 12th, North American technicians began loading hypergols aboard the Apollo for a 23 February launch.23

In mid-February General Phillips, Apollo Program Director, asked KSC to review the shortcomings of the automated checkout. Phillips hoped to use Marshall's breadboard more effectively in the next Saturn IB checkout.24 KSC's response listed 15 general recommendations (and more than 50 specific corrections). One problem area involved configuration differences between the breadboard and LC- 34, e.g., the Cape's telemetry interrupter for the S-IVB stage was apparently quite different from its counterpart at Huntsville. This illustrated a bigger problem: engineering orders accomplished at one site were overlooked at the other. Since the breadboard had not duplicated all of 34's automation, Marshall was unable to assist with certain problems. KSC recommended that Huntsville have more duplication in the breadboard and a capability to respond faster for emergency tests. The launch team accepted much of the blame for the long delays in initializing and reinitializing the computer - which meant the practice of loading the executive routine (in the form of magnetic tapes) into the computer at the start of a work day. Frequently, a failure in the computer hardware would scramble the operating program and force a technician to reinitialize the 110As. The launch team believed the problem would diminish as operators gained confidence in the computers. The report scored the lack of reliable up-to-date documentation (e.g., unit schematics) and concluded:

Possibly the most significant single problem area during AS-201, from complex activation through launch, was the never-ending struggle to obtain Engineering Orders to work changes in Electrical Support Equipment. Very few of the changes in Complex 34 ESE were subject to any level of technical arbitration. Delays and difficulties were primarily simple matters of overcoming the system inertia. 25

It was the hardest thing to do to convince engineers who are used to manual operations that the black box out there, which he cannot fully understand, does a job for him and he will not see the little green lights any longer but the box will do the checking for him. I think this convincing of the engineers was the most complicated task in automation.27

* The emergency detection system alerted the astronauts to a space vehicle failure and initiated escape procedures. The system sensed hundreds of space vehicle functions and provided triple redundancy, i.e., three sensors checked a function to guard against sensor malfunction. While a manual test of this system took 12-14 hours, an automated checkout ran about 20 minutes.

** KSC engineers had coated the circuit boards for protection from the Cape's salt air. During computer operations, heat built up under the coating and cracked some solder joints. Thereafter a wire would open intermittently during operations. The boards were sent back to RCA's California plant after the 201 mission for modification; the "fix" was to place a stress relief eyelet around the wire.