Leaked RAF memo scathing of safety software evaluation

Software "until it can be verified, it has to be considered unpredictable" and a possible cause of a Chinook crash on the Mull of Kintyre in 1994

A leaked draft memo written by one of the RAF's most senior officers has a voice and tone similar to that of a chief executive in the private sector who attacks IT staff for delaying a much-needed new computer system.

The memo criticised the Ministry of Defence's Aeroplane and Armament Experimental Establishment (A&AEE) at Boscombe Down that had been asked by the Ministry to evaluate new safety-critical software for the Chinook Mk2 helicopter.

"I find A&AEE's attitude quite incredible," said the draft memo which was from one of the RAF's most senior officers to one of his colleagues.

The reference date on the memo is 6 June 1994 - four days after the fatal crash of a Chinook Mk2 on the Mull of Kintyre which killed 29 people.

The resentment expressed in the memo reflects the concerns that often exist at the highest levels of an organisation, where the effects of delays in the introduction of a major new system are most keenly felt.

In the RAF's case, the memo makes it clear that senior officers were anxious to introduce a modified Chinook helicopter into service, in order to "maintain a capability". But the IT and airworthiness specialists at Boscombe Down were refusing to approve the Chinook Mk2's new Full Authority Digital Engine Control (Fadec) software and had halted trials flying of the Chinook Mk2 because of their concerns.

Boscombe Down was describing the Fadec software variously as "unverifiable", "not fit for purpose" and "unacceptable". It was always denied that Boscombe Down had declared the software unsafe.

The memo, however, reveals for the first time that Boscombe Down had halted trials flying for the Chinook Mk2 because of "concerns about the safety of the Fadec system". But it went on to attack the A&AEE's ban on trials flying as contrasting sharply with the "considerable efforts being made by the front line to bring the aircraft into service" The A&AEE, said the memo, has an attitude that "does nothing to engender aircrew confidence in the aircraft".

The memo points to a far wider and more serious issue than a simple difference of opinion between the Ministry of Defence and its own independent software and airworthiness assessors.

At the heart of the dispute is the role played by independent scrutineers as arbitrators of the fitness for purpose of critical software.

It raises questions over the value of the A&AEE's advice, and whether its criticisms should be set to one side if they contradict the operational objectives of client bodies such as the MoD.

Malcolm Perks, former IT head of Technology at Rolls-Royce, who helped to develop the company's Fadec systems, believes that there was little point in the MoD asking independent assessors for their views and then discounting any opinions that were not "on-message".

Computer Weekly has, however, pieced together evidence that shows that the MoD has, since the crash, consistently undermined the A&AEE's authority and credibility in letters, Parliamentary answers and in briefings.

Static code analysis

One of the main criticisms of Boscombe Down was that it tested the Fadec software using its "preferred method" of "static code analysis", which was more applicable to the nuclear than the defence industry. Static code analysis is a process of testing parts or the whole of the software, during design, development and final validation, to see that it is free of anomalies and without executing the code.

Boscombe Down described the Fadec as unverifiable, partly because it could not validate the software using static code analysis.

One member of the House of Commons Defence Committee, Mike Hancock MP, was told in an MoD letter last year that "Boscombe Down's preferred method of examination is static code analysis, a system of verification not widely in use but employed in the British nuclear industry."

The letter added: "The Fadec software is not amenable to static code analysis and hence Boscombe Down was not able to verify the software to their [sic] satisfaction... This does not mean that there was any inherent problem with the software."

A month previously, in July 1999, the MoD wrote in a letter that: "Static code analysis does not validate the performance of the software and the Department, therefore, had no requirement for it."

In 1998, the House of Commons Defence Committee was told that the Fadec software could be validated but not by certain scientists within Boscombe Down who insisted on using static code analysis. The committee was told that static code analysis "is a requirement placed by British Nuclear Fuels on the safety of a nuclear system", but Boscombe Down believed it "should be applied to aircraft software".

The Defence Committee concluded that the "failure" of Boscombe Down to give approval to the Fadec software was a "management failure".

And the National Audit Office, based on its MoD briefings, wrote in a report that was due to be discussed by MPs this week that, "the requirement for static code analysis was an internal Boscombe Down policy, not supported by Defence standards."

Computer Weekly has learned, however, that static code analysis is specified in the MoD's own defence standard 00-55. The standard is a benchmark for safety-related software supplied to the MoD.

There are repeated references in 00-55 to the need for static code analysis. One such reference says, "Static analysis... shall be performed on the whole of the source code to verify that the source code is well formed and free of anomalies that would affect the safety of the system." The standard also specifies that the code should be capable of being verified independently by performing dynamic and "static analysis" of the code.

The standard also says that any code not developed to the requirements of 00-55 should, ideally, be reverse engineered to meet its specifications. This would mean retrospective activities that include "static analysis".

Computer Weekly has further learned that the 00-55 standard, as it applied in 1994, at the time of the crash on the Mull of Kintyre, specified static analysis.

Value judgement

All this leaves some IT specialists wondering whether there is an inherent conflict of interest in asking the MoD or the RAF to act as independent judges of whether Boscombe Down was right or wrong in its concerns about Fadec, and whether the software was or was not a contributor to the crash on the Mull of Kintyre.

For if Fadec were ever to be implicated in the crash on the Mull of Kintyre, it might show that the A&AEE's concern was justified all along. It might also show that the hierarchies of the RAF and MoD were wrong to approve the Mk2 despite Boscombe Down's concerns.

More Chinook news