Chinook Mk2: we publish new evidence of computer problems
The lead news item on BBC radio and TV news for much of yesterday referred to computer-related evidence about the Chinook Mk2, of the type that crashed on the Mull of Kintyre in June 1994.
The crash killed all 29 on board including 25 senior police and intelligence officers. The RAF blamed the two dead special forces pilots, Flight Lieutenants Rick Cook and Jonathan Tapper.
BBC’s Online’s headline was: “Chinook crash may have been caused by software faults”.
So what is the new evidence?
It’s in two documents, which are herein published generally for the first time (see below). Thank you to a concerned insider for the information.
The documents were written by the Superintendent and Chief Superintendent at the Ministry of Defence’s Aircraft and Armament Evaluation Establishment [A&AEE], Boscombe Down near Salisbury, which is where engineering and IT experts assess the airworthiness of new and upgraded aircraft.
The Boscombe Down site is now run and managed by Qinetiq.
The documents were written in 1993, several months before the crash of Chinook ZD576. They condemn the Chinook Mk2’s safety-critical fuel control computer system “Fadec” in more direct and stronger terms than criticism of the system in other official documents that have come to light since the accident.
It is clear from the newly-disclosed documents that the Fadec software, which was installed in the Chinook Mk2 was in an appalling state.
It was improved after the crash on the Mull. The MoD’s press office has declined to confirm to me whether, after the crash, the RAF installed on RAF Chinooks the US version of the Fadec software, which had underdone much more testing and was not the same as the UK version.
According to the newly-disclosed documents, the Chinook Mk2 software had, in 1993, a:
– “high density of deficiencies”
– flaw that was “positively dangerous”
The software was of “proven inadequacy” and the amount of testing hours was, in practice, less than 100 hours which was “five orders of magnitude too small”.
One Boscombe Down document said the Fadec software fell “significantly short of the standard required and expected for a safety-critical system”. It said:
“No assurance can be given concerning the fidelity of the software, and hence the pilot’s control of the engine(s) through FADEC cannot be assured”.
Yet, when the pilots of Chinook ZD576 took off from Northern Ireland for a short trip to Scotland on 2 June 1994, they would have relied on the Fadec to control the flow of fuel to the Chinook’s two jet engines.
The software was installed between the pilots and the engines. It was categorised as “safety critical” by Boeing, the Chinook’s manufacturer.
Through their controls, the pilots called for more power and Fadec sent electronic signals to increase the flow of fuel to the engines. The pilots had no direct mechanical control of the engines and the Fadec could not be overridden.
Indeed the Fadec was innovative because both the main channel and the backup – called the “Reversionary Lane” – were controlled entirely by software.
But at the time of the accident on the Mull of Kintyre, Chinook Mk2 pilots were instructed formally not to use the backup channel because it wasn’t working properly, though there were no instructions on what pilots should do if the system itself switched to backup reversionary mode.
Worse, testing of the Chinook Mk2 on the ground and in flight had shown that when the system itself had switched to backup, pilots had lost control of an engine, or engines.
All of this helps to explain why Flight Lieutenants Cook and Tapper had expressed concerns to their fathers about flying the Chinook Mk2.
Campaigners for the families of the pilots point out that Cook and Tapper were blamed unequivocally for a crash that has no certain cause; they couldn’t defend themselves, and they were flying a helicopter that was known to be unsafe.
**
Below is the text of the newly-disclosed documents, with my explanations in brackets.
The first document said a flaw in the Fadec computer was “positively dangerous” and set out the reasons for Boscombe Down’s refusal to recommend a Controller Aircraft Release [CAR]. Without the CAR, the Chinook could not be certified as airworthy – though the RAF went on to issue a CAR, thus ignoring Boscombe Down.
The second document was sent by Boscombe Down to the MoD in London. It confirmed that the first document was Boscombe Down’s formal position.
**
From the Supt of Engineering Systems [at Boscombe Down].
30 Sep 93
CHINOOK MK2 – T55 [engine type] ENGINE FADEC SOFTWARE
1. The hazard analysis of Chinook Mk2 conducted by Boeing identifies the software in the engine FADEC as safety critical and states that “any malfunctions or design errors could have catastrophic effects”. Boeing therefore recommend that the software be written to the standards defined in RTCA/DO-178 for level one software. Investigations at the A&AEE [the MoD’s Aircraft and Armament Evaluation Establishment at Boscombe Down] have not cast any doubt on that analysis.
2. The production standard software (ie code and documentation) has over the past few months been subjected to static code analysis. This is now, and was at the time the software was specified, the normal procedure at A&AEE for verifying safety-critical software for CAR [Controller Aircraft Release – a key step for certifying an aircraft as airworthy]. Although this method involves some formidable mathematics, it cannot reveal anything that could not have been seen in principle, by reading the code and its associated documents. Although only 17% of the code has been analysed [EDS, the code’s assessors, stopped work on the code because of the density of anomalies found] 21 category one and 153 category two anomalies have been revealed. One of these, the reliance on an undocumented and unproved feature of the processor [based on an Intel ASM96 chip] is considered to be positively dangerous.
[EDS in its formal assessment of the Chinook Fadec code reported that it had a particular concern over an anomaly in the backup Reversionary Lane “concerning use of a byte address rather than a word address”. In reply to EDS’s concerns, Hawker Siddeley, which had written the Fadec software, gave an assurance that the code behaved correctly. EDS was not satisfied with Hawker Siddeley’s assurances. In its report on the Fadec software, EDS said: “Correct operation (of the Fadec) depends on an undocumented feature of the Intel ASM 96 microcomputer. EDS-Scicon’s concern is that a change in the mask or process of the ASM chip at some point in the future may cause incorrect operation of the FADEC.]
3. Examination of the documentation shows that the procedures selected for controlling and recording this software’s development were good practice at the time. Had they been properly implemented, they should have resulted in software that is fit for safety-critical applications. The traceability study, however, revealed 35 category one and 39 category two anomalies in the documentation, showing that implementation was not good.
4. The density of anomalies (code and documentation) is large; one category one and three to five category two would be more reasonable. Worse, the nature of the software is such that corrective action is considered to be impractical and independent verification is virtually impossible. The assessment has been abandoned, rather than completed, on the grounds that the software is of proven inadequacy. The results are in fact similar to those derived from a study of the development standard which was terminated in September 1990 for similar reasons.
5. Errors in software do not occur randomly and are therefore not amenable to prediction using statistical techniques such as sampling. Neither is it possible to associate a numerical estimate of risk with the use of software. Even were it technically feasible, the number of variations possible is so great that a sample size that could be tested would not produce even the smallest level of assurance. Nevertheless, we have examined the evidence of practical testing and consider the amount to be inadequate even for hardware.
6. The total running time of all versions of FADEC is less than 100,000 hours (rig and engine) … Nearly all of the testing was done using development versions of the software rather than the version we have now. Some of the versions tested, including that flown at A&AEE in 1990, are not even direct ancestors of the current version. The older testing must therefore be disqualified, leaving the relevant testing at less than 100 hours, ie more than five orders of magnitude too small, even for hardware.
7. Conclusions. Our latest considerations do not necessitate a change to the conclusions of Reference A that, in our opinion, the software quality falls significantly short of the standard required and expected for a safety-critical system, and as such cannot be independently verified. From what we know, there is no realistic hope that further studies will change this conclusion, certainly flight trials will not.
a) The density of deficiencies is so high that the software is unintelligible.
b) Because of the density of deficiencies, it would be impractical to maintain the software as re-verification of all the software would be required after every change.
c) No assurance can be given concerning the fidelity of the software, and hence the pilot’s control of the engine(s) through FADEC cannot be assured.
d) The standard of engineering is demonstrably not that to be expected intended for the purpose of controlling a safety critical function in an aircraft.
e) Although it is never possible to quantify the risk associated with the use of software, in this case it is obvious that available measures to ensure the quality of engineering for safety-critical software, were not employed.
f) CAR [Controller Aircraft Release] for Chinook Mk2 with this version of software in FADEC cannot be recommended.
8. Recommendations. Since it appears impractical to retain the hydro-mechanical system used in Chinook Mk1, urgent consideration should be given to re-writing the FADEC software.
Supt of Engineering Systems
Distribution:
TMPO Chinook
S/L RWE
S/L EAS
**
Below is the text of the second newly-disclosed document. It confirmed that the Superintendent’s memo of 30 September 1993 was Boscombe Down’s “formal position”.
To: Director Helicopters Projects
1-13 St Giles High Street
London
From: Chief Superintendent, Aircraft and Armament Evaluation Establishment, Boscombe Down, Salisbury
12 October 2003
CHINOOK MK 2 – CAR [Controller Aircraft Release] for T55 [engine type] FADEC
Reference [Boscombe Down’s letter of 30 September 1993]
1. You will be aware of the work undertaken by A&AEE in attempting to verify the software used in the T55 engine FADEC. Please find attached copy of reference which is an interim report stating our formal position which has in effect already been relayed to your staff.
2. I am well aware that our inability to recommend CAR for the FADEC will cause you a problem, particularly as it appears impractical to revert to the hydro-mechanical system used in the Mk1 aircraft. I can assure you, however, that whilst our flight trials are underway, we will continue to seek ways of keeping any risk associated with FADEC malfunction to a minimum with the intention of providing you with such advice for consideration during your deliberation on CAR.
3. Should you accept our recommendation to re-write the software, we strongly recommend that it is done with some urgency, and introduced into service at the earliest opportunity when the rate and demands of flying should be least, and exposure of the aircraft to risk is least. (This of course assumes a release to service for the existing software is provided in some form).
**
Links:
Chinook computer flaw was “positively dangerous” says newly-disclosed MoD documents – ComputerWeekly.com
Computer Weekly’s 140-page report into Chinook software problems – RAF Justice
Critical internal memo on Chinook software problems – Military-quotes.com forum
RAF knew software in fatal crash was dangerous – Irish Times
Sir Malcolm Rifkind questions Chinook crash finding in light of new evidence – PoliticsHome
Please shut up about the Mull of Kintyre crash – The Register
Did MoD deceive RAF Board of Inquiry? – ComputerWeekly.com
Criticism of the MoD after disclosure of new evidence – Angus Robertson MP
Chinook crash: software failure or pilot error? – ComputerWeekly.com
Computer Weekly briefing to MPs on Chinook Mk2 – Parliament’s website
Discussion of new Chinook evidence on Pilots Rumour Network – Pprune
174 glitches in doomed Chinook software – Myjournal [Scotsman]
Mod rejects calls to reopen inquiry into Crash crash – The Times
Chinook crash may have been caused by software faults – BBC Online
Software linked to Chinook tragedy – MoD news
Magazine disputes Chinook crash cause – BBC