STAMP Accident Model for Safety Engineering :
A Critical Analysis

Robert Setiadi, S.Kom, M.SoftSysEng

1. Introduction

Unlike other "common" systems, safety critical systems need special requirements techniques to deal with. When system failure can lead to unacceptable consequences such as lost of lives or properties, then the system is categorized as safety critical system [1, 2]. For many years, lots of studies and preventive preparations had been performed to avoid any accidents caused by system failure. Those preventive actions, however, still failed to identify some potential hazards of a safety critical system.

Accident is defined as "an event or sequence of events leading to harm, that is, death, injury, environmental damage, or financial lost". Hazard is defined as the precondition for an accident. [3] The aim of safety engineering study is to minimize (or to eliminate) potential hazards of a system. Until today, there is no "super technique" that can produce state-of-the-art hazard analysis for all kind of systems. While most researchers try to improve the existing techniques, some started to create a different and better approach.

Will a new analysis technique answer safety engineering challenge and provide us with better safety systems? STAMP accident model (Systems-Theoretic Accident Model and Processes) proposed by Nancy G. Leveson [4] has brought a promising future in the area of safety engineering. This technique, however, still need more work and better communication in order to get more support from real-world society.

This paper will provide a literature study on safety critical system analysis requirements engineering. The following sections present semi-historical evolution of Leveson's ideas and other accident analysis approaches and techniques. Some case studies and challenges on STAMP, including Peter B. Ladkin's WBA (Why-Because Analysis) technique [5] will be briefly discussed to evaluate this new approach of safety-critical system requirements.

2. Safety Requirements Engineering

Safety critical systems – such as Computer Aided Dispatch Systems, Flight Control Systems, Medical Systems, Space Shuttle Systems, etc – need different approach of requirements engineering because those systems should not fail under any circumstances. More careful analysis needs to be conducted to ensure that those systems are safe and reliable. [3]

There are three basic types of hazard analysis (1) exploratory technique e.g. HAZOP (Hazard Operability Analysis), (2) causal technique e.g. FTA (Fault Tree Analysis), and (3) consequence technique e.g. FMEA (Failure Modes and Effects Analysis) [3]. These techniques can be used to support one another to produce a stronger (and more complete) technique of safety engineering.

The basic principle of HAZOP technique is that "a problem can only arise when there is some deviation from the intent of the system as represented by the model under review". Its study provides three models (1) formal model, (2) algorithmic model, and (3) causal model [6]. Developed in the 1950s, HAZOP became widely accepted due to its successful application at British chemical industry [7]. This approach, however, is not naturally fits modern technology since it is basically rely on manual analysis, evaluate only the deviation between outcome and expected behavior, and analyze the problem based on flow-based point of view [7].

One popular language in HAZOP formal model is Z language. Properties and behavior of systems are specified using Z language in set-based notation. Guidewords such as "more", "less", "early" and "late" are used to describe cause and effect of system failures. [3, 6, 8]

HAZOP causal model illustrates system failure in diagrams of multiple causes, single effect and multiple consequences. As comparison, causal model of FTA relates multiple causes to single consequence while causal model of FMEA relates single cause with multiple consequences. [6]

3. Hazard Analysis using Conventional Models

Conventional methods of accident analysis use discrete failure events to identify the cause of accident [9]. The idea is that an accident is caused by a root cause. The aim of an analysis is to find out which component of the system can potentially become the root cause of an accident. [6, 9]

This conventional approach uses event-chain models to provide the process of accident analysis [10]. These models generally "explain accidents in terms of multiple failure events, sequenced as a forward chain over time" [11, 12]. There are many analysis techniques based on this event-chain point of view. Fault tree analysis, failure modes and effects analysis, event trees and probabilistic risk assessment are some of the popular techniques which have been used by many engineers, and trusted as the "best" way of safety and reliability engineering for decades [11].

As a result of this approach, safety and reliability engineering had been actually focusing more on the reliability side rather than safety side. Conventional experts believe that in order to prevent an accident, one must break the chain of events which cause it. This idea is the basic theory behind some reliability techniques such as redundancy, overdesign and safety margins. [11]

In reality, conventional accident analysis methods are often failed to handle complex safety critical systems where accidents can emerge not from few components only, but from the interactions between system components. Leveson [1, 13] and her colleagues had proposed a new approach named STAMP to perform safety analysis on complex systems.

4. Leveson's Safety Engineering Research

4.1. Early Improvements

Nancy G. Leveson – the creator of STAMP – is a professor of Aeronautics and Astronautics Dept. and also Professor of Engineering Systems at MIT. Her main research areas are system safety, software safety, software and system engineering and human-computer interaction. Having a lot of real experience working with NASA and working on several case studies, Leveson and her students produced many publications in accident model and software safety areas [1]. In 1995, she wrote her first book "Safeware: System Safety and Computers" [14] about Safeware methodology. Arguably, this methodology is the early conceptualization of STAMP.

The first significant "rejection" to the conventional event-chain models was initiated by a group of European researchers lead by Prof. Jens Rasmussen. STAMP was then developed as a further step of developing pure system-based accident model, departing from the basic concept of system safety. [11]

Earlier ideas of STAMP started to appear in her papers long before she formally published STAMP technique. In her paper on Therac-25 accident investigation (1993) she pointed out the importance to distinguish between safety and reliability, the need of system self-check to detect and handle errors, also the importance of seeing software as a dynamic critical part of a system. [15]

4.2. Safeware Methodology

In 1997 Leveson published a paper about Software Deviation Analysis (SDA). SDA is one of the Safeware hazard analysis techniques written in her book. Her theory took the strong characteristics of HAZOP technique – a hazard analysis technique from chemical industries which then applied to software safety – such as guidewords, deviations, exploratory analysis and systems engineering approach to build an automated procedure of deviation analysis. [7]

SDA procedure started from causality diagrams from analyst, calculate deviation formulas and return a set of scenarios. As mentioned above, SDA took strong ideas of HAZOP and implemented them into its procedure while removing the weakness with some other details more suitable for software safety study. [7]

Safeware methodology is a safety analysis methodology for complex computer-based system. Its techniques include (1) management structures and procedures, (2) system hazard analysis, (3) software hazard analysis, (4) requirements modeling, (5) completeness and safety analysis, and (6) special software design techniques such as (6a) human-machine interaction, (6b) verification, (6c) operational feedback, and (6d) change analysis. [16]

Leveson also introduced her own specification language and tools for safety engineering named SpecTRM (Specification Tools and Requirements Methodology). It is basically a development environment to design a system with integrated safety and hazard analysis. In many research, she used both Safeware methodology and SpecTRM toolset to perform her analysis over a system. [17]

A good example of the application of Safeware methodology and SpecTRM toolset is the case study of CTAS (Center TRACON Automation System) software (NASA's air traffic control system) in December 1997. Leveson and her team demonstrated the performance of Safeware and SpecTRM using the following processes: [16]

• Preliminary Hazard Analysis
Her approach of PHA includes making a hazard list, generating fault trees analysis, and defining safety requirements and constraints.

• Modeling
The aim of formal modeling is to discover undetected hazards when only high-level design is put into account. Leveson had designed some formal modeling language such as (1) RSML (Requirements State Machine Language) which was applied in her former projects, (2) SpecTRM-RL which is an improved version of RSML with more emphasis on readability, black box models and special constraints enforcement.

• Controller Task Analysis
This process models controller tasks using SpecTRM-RL modeling language.

• Completeness and Consistency Analysis
Based on the model already built, the next process is to analyze the completeness and consistency of that model. Completeness analysis is important to find out incompleteness with respect to process state, input and output, human-computer interaction, trigger-event, output specification, output to trigger event relationship and transition completeness.

• Simulation and Animation
Simulation and animation are important to help designers and analysts to get the visualized picture of the system.

• State Machine Hazard Analysis
This analysis uses state machine to perform top-down and backward test to obtain how hazardous states can be reached. If a hazardous state can be reached, two possible actions can be taken, either to remove the path to that state, or making a controller.

• Deviation Analysis
Hardware-related deviation analysis is performed by FMEA and FMECA (Failure Models and Effects Critically Analysis). Software-related deviation analysis is performed by SDA.

• Mode Confusion Analysis
This process includes analysis on interface interpretation errors, inconsistent behavior, indirect mode changes, operator authority limitation issue, unintended side effects and lack of appropriate feedback.

• Human Factor Analysis
Human factor analysis is conducted by evaluating the effect of task design on human performance with respect to safety. Issues like communication, situation awareness, vigilance and workload are given serious attention in this analysis.

• Operation Research Modeling and Analysis
The purpose of this research is to evaluate the impacts of different operational strategies (such as scheduling algorithm) to the trade-off between efficiency and safety.

• Intent Specifications
Intent specifications are the system specifications in SpecTRM. Its goals are to create and maintain good communication between disciplines, ensure traceability, support human problem solving and integrate formal and informal specifications.

In her Safeware methodology, Leveson categorized accidents into two types (1) Component-Failure Accidents and (2) System Accidents [17]. While many other hazard analysis techniques can handle component-failure accidents reasonably well; the second category (system accidents) needs more attention with the rapid development of digital technology [17, 18]. Leveson continuously improved her Safeware methodology and SpecTRM toolset until 2001 [18, 19, 20]. After that she introduced a new accident model based on systems theory in 2002 [21, 22].

4.3. Systems-Theoretic Accident Model and Processes (STAMP)

Leveson's first paper about new accident model based on systems theory was written in 2002 and published in The International Conference of the System Safety Society, Denver [21]. In the same conference, she also published a sample application of her new model in the case of friendly fire accident in the Iraqi-No-Fly-Zone 14 April 1994 [22]. No specific name had been given for this new model at this point.

Later in 2002, Leveson named her new model "STAMP" (Systems-Theoretic Accident Model and Processes) and her new accident analysis based on STAMP as STPA (STAMP-based analysis) [23]. This new STAMP model evolved rapidly until Leveson wrote an article "A New Accident Model for Engineering Safer Systems" [4] in 2003 (which published later in 2004), which she called as the answer of "what the model really is" in a public mailing list from The University of York [24]. There she also mentioned that the first idea of STAMP was actually started in 2000, two years before the publication of the first paper [24].

Conventional event-chain models explain an accident as a linear sequence of events. One event leads to another and causes the accidents in the end. To prevent an accident, the first event in the sequence (the root cause) of that accident must be found; some efforts need to be made to break that chain. This approach focuses more on system components in the physical form, and has limited capability to explain (1) social and organizational factors, (2) system and software errors, (3) human errors and (4) adaptation as potential cause of an accident. [4]

In STAMP, systems are no longer viewed "simply" as a set of components. Accidents occur not simply because of a component failed to perform its function. A system continuously changes as a respond to information (input) and the environment, leading to dynamic "balanced" state of a system. Interactions between components can lead to accidents if the system reaches imbalance (hazardous) state and inappropriate control was taken. [4]

STAMP model believes that to prevent accidents, system need to be controlled so that no system constraint is violated. Instead of thinking in the term of "events", STAMP uses "constraints" as its very basic concept. Hazard is seen as an unsafe behavior of a system. Accidents might happen if control fails to enforce constraints on system behavior, or if a system requirement (and design) fails to identify a constraint. [4, 25, 26]

Accident model based on systems-theory is developed using these basic ideas: [4, 10]

• Emergence and Hierarchy

In systems theory, complex system is expressed in hierarchical level where each level is more complex than the previous level. Each level contains system components and their interactions. Levels are differentiated using emergent property – the property of a system which does not exists at the lower level – which separates one level with another.

When applied to STAMP, safety becomes the emergent property. Every time interactions between system components raise a new safety constraint, a new level is created. A system component will not be evaluated alone to assess its safety because it might be perfectly safe in a level but not safe when interactions with other system components are considered.

Relationships between levels of complexity are analyzed using hierarchy theory. Emergent safety properties in every level must follow certain system behaviors, and therefore they have to follow some constraints attached to them. When all these constraints are fulfilled, no hazardous state will be reached.

• Communication and Control

As a system dynamically changes, a control is needed to keep all constraints are imposed. This control holds the rule to keep the balance of a system and therefore will maintain communication between system components. Components in a closed system might change state because of inputs; components in an open system can change state because of inputs and environment.

STAMP considers safety as a control problem. A safe system must continue to operate safely after a system reacts to its environment and changes dynamically. In this case, adaptation and feedback become the important issues. Downward communication provides reference channel where constraints are imposed in the level below. Upward communication provides measuring channel where feedback is provided to the level above.

According to systems theory, accidents occur because (1) hazards and safety constraints are not identified, (2) the controllers fails to maintain safety constraints, (3) lack of coordination between controllers, (4) the process models becomes inconsistent with system process and (5) controllers can not retrieve system state because of the lack of feedback. [9, 10, 11]

One main difference of STAMP compared to conventional hazard analysis is that STAMP considers the following system components: hardware, software, people, technical organizational, societal and organizational structures, engineering activities and dynamic factors in modern complex systems. Instead of focusing on certain system components, STAMP analyzes those components in the term of their interactions to each other. [9, 10]

Controllers can be human controllers and automated controllers. It is important to ensure that mental model of human controller and design model of automated controller can work safely [4]. Reference channel and measuring channel provide useful ways to maintain the controlling process effective.

All constraints defined by safety engineers need to be communicated clearly and systematically to system developers. Inadequate communication of constraints may cause a deviation between a model of a system and the actual process, and therefore will lead to accidents. [4]

4.4. STAMP Case Studies

Some case study papers had been published by Leveson and her team to provide good examples on their new approach on safety engineering. Not all published examples are discussed in this paper.

• Walkerton water contamination case [10, 27]

Walkerton is a small town in Ontario, Canada. There was an accident in May 2000 when E.coli and Campylobacter jejuni bacteria contaminated the water system, causing about half of the town people become ill and seven died. STAMP approach was used to analyze this accident and an investigation report was published in 2003.

Water system in Walkerton was maintained by a local commission named Walkerton Public Utilities Commission (WPUC). In May 2000 they have three sources of water: wells 5, 6 and 7. Water was treated using chlorine before pumped from the wells into the distribution system.

Koebel, the general manager of WPUC operated well 7 without chlorination from May 15 to May 17. On May 15, A&L Labs, a private test lab advised Koebel that the water was tested positive containing E.coli. Illness spread in the community within the next few days. Instead or reporting the test result Koebel took his own "solution" to the problem by superchlorinating the water system because he tried to hide the fact that he was running well 7 without chlorination for 3 days.

When government unit investigated the accidents, Koebel claimed that this E.coli spreading had no connection with the water system. Providing fake reports to the Ministry of Environment, he tried to conceal his "mistake".

It was in May 21 when the government health unit realized that E.coli bacteria were found positive in the water system. Boil water advisory was immediately announced, and further risk can be reduced. Later investigation then found that the E.coli bacteria were not actually coming from untreated water in well 7, but coming from well 5. It was the heavy rain from May 8 to May 12 that carried the bacteria from nearby farms to well 5.

Using systems-theoretic explanation, the safety constraints in this system are (1) water quality should not be compromised and (2) public health measures must reduce risk exposure if water quality is compromised. Instead of putting all the blame on Koebel, accident analysis using STAMP came up with the following issues:
• The operators of the WPUC (in this case is Koebel brothers) did not intentionally harm the public safety because they believe untreated water is safe. They are lack of training and knowledge on water safety.
• Local residents in Walkerton even pressed WPUC to reduce the level of chlorination because they dislike the taste of chlorinated water.
• The government local health unit is the first level control of the system.
• The current practice on water safety control is already different from the designed control because in 1996 the government privatized water testing laboratories. As result, the test results were no longer automatically shared with the government agencies.
• In the privatization of water testing laboratories, The Ministry of Environment only sent guidance documents to those who requested. MOE did not inform the laboratories any guidance.
• Previous warnings from water testing labs that surface water has met the water in wells were ignored.
• Previous warning from a Health Canada study that Walkerton is located in high-risk areas for E.coli infection was ignored.
• The lack of protocols in some government agencies.

Hazard analysis using STAMP identified that the water safety control structure in Ontario had improved over time by the introduction of operator certification and other requirements added in 1994. However, some negative progress also took place. Feedback to Ministry of Health and Ministry of Environment was reduced by the reduction of inspections.

The privatization of water testing laboratories did not reduce safety. But as water testing laboratories are important components of water safety system, changes in one component would bring different interactions with other components. In this case, no adjustments were taken; no controllers were introduced with new constraints; and the controllers (WPUC operators, government units) can not provide optimal prevention of constraint violation.

Changes on the environment also may affect the constraints of a system. Some constraints may need to be changed and some new constraints may need to be introduced. In Walkerton case, the rapid increase of factory farm and animal waste changes the environment, but the safety control system was not re-evaluated to verify its capability in dealing with new environment.

• Titan/Centaur/Milstar loss case study [26, 28, 29]

Titan IV B-32/Centaur TC-14/Milstar-3 was launched in April 1999 to put Milstar-3 satellite in its orbit. The mission was completely failed since Milstar-3 ended up in an unusable low elliptical orbit instead of the intended one.

Investigation board analyzing this accident concluded that the root causes of this accident are "inadequate software development, testing and quality assurance" [26]. The software failed to detect a human error and correct it.

On the launching day (30 April 1999), the software already reported all zero roll rate. No one is responsible to monitor the software report, and no further actions were taken. Only hardware checks were conducted before the launch time, but the software was assumed to be correct because software testing already remove all software errors.

In the software development period, due to budget reduction, no person from civil service or military personnel was assigned to support (and monitor) the work of software development. Since the software already used in the past and had no problems, the software was assumed to be safe and stable.

Conventional event-chain models produced only the "software development, testing and quality assurance" as the "root cause" of the accident. It failed to identify the reason behind why such software can be used in a very expensive mission without anyone being responsible to monitor its report on the final day.

The largest number of STAMP examples comes from spacecraft accidents. This is understandable considering Leveson's expertise in this area and her close relation with NASA. Beside this Titan/Centaur/Milstar case, STAMP analysis was also been applied to investigate ARIANE 5 accident in 1996 [12], SOHO (SOlar Heliospheric Observatory) spacecraft loss in 1998 [18, 30], Mars Climate Orbiter loss in 1999 [31] and Mars Polar Lander destruction in 2000 [32].

4.5. Criticism and Challenges on STAMP

STAMP, as a new technique, triggered different reactions from other researchers. Criticism on this new approach appears on January 2003 when Johnson and Holloway published a result paper of their experiment using this new technique. Their paper gave mainly positive result by acknowledge the significant benefits from using STAMP to "correct" chain-of-events model. Aside from the positive feedback, Johnson and Holloway also pointed out the need of "documenting the relationship between constraint 'flaws' and any subsequent recommendations". [30]

Peter B. Ladkin wrote a controversial article on STAMP in New Scientist magazine [33]. Most of his criticisms, however, were answered by Leveson as the lack of understanding of STAMP model and its idea [24, 34]. Ladkin himself is a professor in Computer Networks and Distributed Systems at the University of Bielefeld in Germany. He is the creator of another safety analysis theory named WBA (Why-Because Analysis) [5, 35, 36].

WBA is defined by Ladkin as "a rigorous technique for causally analyzing the behavior of complex technical and socio-technical systems". This technique analyzes accident using causal factors; building a graph named WB-Graph (WBG) to show those causal connections. In WBA homepage, Ladkin claimed that WBA is "the only accident analysis method with such a formal consistency and completeness check". [36]

A comparison of STAMP and WBA techniques was published in April 2005 by William Greenwell. While introducing his approach on safety-related digital system failures analysis named Pandora, Greenwell addressed STAMP's strength as the capability to "analyze interactions between subsystems" and WBA's strength as the capability to "produce a rigorous proof of causal arguments". [37]

5. Critical Review

Is a total new approach needed to create better safety-critical systems? Or do we only need to improve current (conventional) techniques on safety engineering to make them better?

The advancement of technology has brought new changes that impact human ways of life, and therefore also impact the systems created by humans. Many decades ago, the main issue of "safety" is simply to ensure that all system components do not fail. Reliability was emerged to be more dominant than the real meaning of safety. Hardware endurance improvements, duplication and many other techniques were developed to ensure that system will not go wrong. [9, 10]

As computer technology becomes more popular nowadays, many systems started to have digital components and become more complex. Software becomes inevitable part of most systems. A new problem was emerged because reliable software can be unsafe and safe software can be not reliable. The concept of reliability and safety often brought confusion and not many people realized that they are actually two different issues.

While hardware components become more reliable, increasing software can not be done in the same way as increasing hardware reliability. Analyzing safety is different from analyzing reliability. Software development, as well as modern system development, is very sensitive to other issues in the system such as social and organizational factors, interactions with other system components, environment, adaptation and feedback and the constraints need to be imposed.

Analysis techniques using event-chain models and techniques are no longer the ideal solution for safety engineering. No matter how far one can make improvements on those techniques, they still based on the conventional ideas of causes, events, effects and consequences. Therefore these techniques' ability to detect root causes of hazards is limited by its "linear" chains.

STAMP technique has provided safety engineering research with a fresh idea. Considering safety analysis from systems-theory enable us to see a system in hierarchical point of view. Dealing with safety constraints instead of events gives us a broader view of a system to enable us to capture deeper aspects of an accident. The adaptation and feedback mechanism ensures that a system is able to maintain safe operations after changes in the environment. [4]

In many study cases, accidents often occur because the current practice of the systems are no longer match their original designs. Changes in environment are ignored and the current practices of the systems were not adjusted. System controllers (human and automated) often do not properly communicate, and produce poor coordination.

Negative criticisms on STAMP are considered to be "less relevant" due to the fact that Ladkin's criticisms towards Leveson's ideas already started from 1997 (Safeware methodology) and tends to become more personal "dislike" rather than academic feedback. Most of those criticisms had been answered by Leveson as misinterpretation of the idea of STAMP. [38, 39, 40, 41, 42, 43, 24, 34, 44]

However, it is also notable that STAMP still needs further improvements to become a more complete methodology. More examples would be very useful to help engineers to apply this technique. Since the technique itself was still evolving in its earlier publications, people reading only those early publications might have incomplete idea about STAMP. Better structured explanation would also help this new approach to be widely accepted and applied in real-world current problems (this will be materialized by Leveson's upcoming book on STAMP [45]).

6. Conclusions

A new approach is needed to answer the urgent demand on safety requirements engineering. This paper has showed that existing event-chain techniques are no longer the ideal solution to accident analysis. The characteristic of modern systems themselves have changed to adapt with emerging digital technology. This new characteristics need a new approach to deal with safety issues.

To ensure safety, one must not consider only individual components of a system. Interactions between components, safety constraints, system controllers, human factors, social and organizational influences, dynamic environment changes and many other aspects need to be raised into consideration when making safety requirements and design a safety-critical system.

Instead of seeing a system as a static entity, a system needs to be treated as dynamic balance of its components. To ensure safety, system controllers (human and automated) must maintain system safety constraints and avoid any hazardous states.

References

1. N. G. Leveson, Nancy leveson's home page at mit, cited: November, 2005, http://sunnyday.mit.edu/

2. J. C. Knight, Safety critical systems: Challenges and directions, Proceedings of the 24th International Conference on Software Engineering, ACM Press, New York, NY, USA, 2002.

3. E. Kazmierczak, 433-646 lecture notes, chapter 7, The University of Melbourne (2005).

4. N. G. Leveson, A new accident model for engineering safer systems, Safety Science 42 (Apr 2004), no. 4.

5. P. B. Ladkin, A quick introduction to why-because analysis, University of Bielefeld (1999).

6. P. Fenelon and B. Hebbron, Applying hazop to software engineering models, Risk Management And Critical Protective Systems (1994).

7. J. D. Reese and N. G. Leveson, Software deviation analysis: A "safeware" technique, AIChe 31st Annual Loss Prevention Symposium, Houston, TX (Mar 1997).

8. E. Kazmierczak, M. Winikoff, P. Dart and L. Sterling, Verifying model oriented specifications through animation, Software Engineering Conference, 1998, pp. 254-261.

9. N. Dulac and N. G. Leveson, An approach to design for safety in complex systems, International Conference on System Engineering (INCOSE '04), Toulouse (Jun 2004).

10. N. G. Leveson, M. Daouk, N. Dulac and K. Marais, A systems theoretic approach to safety engineering, (Oct 2003).

11. L. G. Leveson, Safety in integrated systems health engineering and management, NASA Ames Integrated System Health Engineering and Management Forum (Nov 2005).

12. J. L. Lions, "Ariane 5: Flight 501 failure," ARIANE 5 Inquiry Board, 1996.

13. Mit complex systems research laboratory papers, (2005), cited: November, 2005, http://sunnyday.mit.edu/papers.html

14. N. G. Leveson, Safeware: System safety and computers, Addison Wesley, 1995.

15. ---, The therac-25 accidents, (Jul 1993).

16. N. G. Leveson, L. Alfaro, C. Alvarado, M. Brown, E. B. Hunt, M. Jaffe, S. Joslyn, D. Pinnel, J. Reese, J. Samarziya, S. Sandys, A. Shaw and Z. Zabinsky, Demonstration of a safety analysis on a complex system, Software Engineering Laboratory Workshop, NASA Goddard (Dec 1997).

17. N. G. Leveson, System safety in computer-controlled automotive systems, SAE Congress (Mar 2000).

18. K. Weiss, N. G. Leveson, K. Lundqvist, N. Farid and M. Stringfellow, An analysis of causation in aerospace accidents, Space 2001, Albuquerque, New Mexico (Aug 2001).

19. M. Katahira and N. G. Leveson, Use of spectrm in space applications, The 19th International System Safety Conference, Huntsville, Alabama (Sep 2001).

20. N. G. Leveson, M. d. Villepin, M. Daouk, J. Bellingham, J. Srinivasan, N. Neogi, E. Bachelder, N. Pilon and G. Flynn, A safety and human-centered approach to developing new air traffic management tools, ATM 2001, Albuquerque NM (Dec 2001).

21. N. G. Leveson, A new foundation for system safety, International Conference of the System Safety Society, 2002.

22. ---, The analysis of a friendly fire accident using a systems model of accidents, International Conference of the System Safety Society, 2002.

23. ---, "Model-based analysis of socio-technical risk," Technical Report, Engineering Systems Division, Massachusetts Institute of Technology, Jun 2002.

24. P. B. Ladkin, N. G. Leveson and M. Thomas, Safety-critical mailing list archive 2003: Comments on stamp, (2003), cited: November, 2005, http://www.cs.york.ac.uk/hise/safety-critical-archive/2003/

25. N. G. Leveson, A new approach to hazard analysis for complex systems, International Conference of the System Safety Society, Ottawa (Aug 2003).

26. ---, The role of software in spacecraft accidents, the AIAA Journal of Spacecraft and Rockets 41 (Jul 2004), no. 4.

27. N. G. Leveson, M. Daouk, N. Dulac and K. Marais, Applying stamp in accident analysis, Workshop on Investigation and Reporting of Incidents and Accidents (IRIA) (Sep 2003).

28. N. G. Leveson, A systems-theoretic approach to safety in software-intensive systems, IEEE Trans. on Dependable and Secure Computing (Jan 2005).

29. N. G. Leveson and N. Dulac, Safety and risk driven design in complex systems of systems, The 1st NASA/AIAA Space Exploration Conference, Orlando (Feb 2005).

30. C. Johnson and C. M. Holloway, The esa/nasa soho mission interruption: Using the stamp accident analysis technique for a software related 'mishap', (2003).

31. D. Isbell and D. Savage, "Mars climate orbiter failure board releases report, numerous nasa actions underway in response," 1999.

32. "Mars program independent assessment team summary report," 2000. http://sunnyday.mit.edu/accidents/mpiat_summary.pdf

33. P. B. Ladkin, "Comments on stamp," New Scientist, 2003.

34. D. Crocker, A. C. Rao, B. Wichmann, D. Crocker, M. Thomas, N. G. Leveson, S. Crook-Dawkins and P. B. Ladkin, Safety-critical mailing list archive 2004: Automatic code generation in safety-critical software development, (Jan 2004), cited: November, 2005, http://www.cs.york.ac.uk/hise/safety-critical-archive/2004

35. P. B. Ladkin, "Report on the accident to airbus a320-211 aircraft in warsaw on 14 september 1993: Main commission aircraft accident investigation warsaw," University of Bielefeld, Faculty of Technology, 1994.

36. ---, The why-because analysis homepage, (2005), cited: November, 2005, http://www.rvs.uni-bielefeld.de/research/WBA/

37. W. S. Greenwell, "Pandora: An approach to analyzing safety-related digital system failures," April 2005.

38. Safety-critical mailing list archive 1997, (1997), cited: November, 2005, http://www.cs.york.ac.uk/hise/safety-critical-archive/1997

39. Safety-critical mailing list archive 1998, (1998), cited: November, 2005, http://www.cs.york.ac.uk/hise/safety-critical-archive/1998

40. Safety-critical mailing list archive 1999, (1999), cited: November, 2005, http://www.cs.york.ac.uk/hise/safety-critical-archive/1999

41. Safety-critical mailing list archive 2000, (2000), cited: November, 2005, http://www.cs.york.ac.uk/hise/safety-critical-archive/2000

42. Safety-critical mailing list archive 2001, (2001), cited: November, 2005, http://www.cs.york.ac.uk/hise/safety-critical-archive/2001

43. Safety-critical mailing list archive 2002, (2002), cited: November, 2005, http://www.cs.york.ac.uk/hise/safety-critical-archive/2002

44. Safety-critical mailing list archive 2005, (2005), cited: November, 2005, http://www.cs.york.ac.uk/hise/safety-critical-archive/2005

45. N. G. Leveson, A new approach to system safety engineering [new book draft], 2005.

Any comments, questions, or suggestions can be sent to robertsetiadi@gmail.com