##,. ####,. #######,. ######,. ##, #####,. #,.#,. ##,. ##, ##,. ##,. ##,. ##, ##,. #,. ##,.##,. ##,. ##,. ##,. ##,. ##, ##,. ##,. ##,. ##,. #######,. ##,. ##,. ##, ##, ########,. ##,. #, ##,. ##,. ##,. ##, .###, ##,. ##,. ##,.. #, ##,. ##,. ##,. ##, ##,.. ##, ##,. ##,. ######, #######,. #######, ##, ######, ## http://www.agedis.de #############.. ##.. +===================================================+ +======= AGEDIS Newsletter =======+ +======= March 2002 =======+ +===================================================+ The AGEDIS NEWSLETTER is E-mailed quarterly to subscribers worldwide to provide information on the AGEDIS EC project in particular and on software testing and software test generation in general. Permission to copy and/or re-distribute is granted, and secondary circulation is encouraged by recipients of the AGEDIS newsletter provided that the entire document/file is kept intact and this complete copyright notice appears with it in all copies. Information on how to subscribe or unsubscribe can be found at the end of this issue. (c) Copyright 2002 by imbus AG, Germany. For best viewing quality we recommend the use of non-proportional fonts, like Courier in your email displayer. ======================================================================== o On Test Directives by Alessandra Cavarra, Jim Davies o How the AGEDIS compiler will be tested by Bernd Mattern o About us: Using AGEDIS at IBM Hursley by Ian Craggs o AGEDIS at conferences: Conference Report by Bernd Mattern o AGEDIS newsletter Article Submission, Subscription Information ======================================================================== On Test Directives by Alessandra Cavarra, Jim Davies (Oxford University) The first AGEDIS language specification dealt with the language of models; six months later, with the prototype tools taking shape, we need a new language: a language of tests. In this article, we describe certain aspects of that language, and explain how they can be realised in UML. Our purpose is two-fold: to present the terminology of _test directives_; and to invite comments, criticism, and requests for elaboration. 1. Test Directives We use the term 'test directive' to denote the collection of information that, when combined with the system model, defines the test suite that will be generated. A test directive is a formal expression of some testing requirement; a user might be expected to define several test directives for a single system model. There are three aspects to a test directive: a test purpose, a set of test constraints, and a set of coverage criteria. 2. Test Purposes A test purpose is a description of a pattern of behaviour. As with the patterns of behaviour in the model itself, it can be expressed as a UML state diagram. There are three illuminating differences: * The test purpose is a description of _system_ behaviour, so we have events and actions from different classes appearing in the same state diagram. * The trigger and action expressions that decorate transitions can include regular expressions. They do _not_ represent processing of events by the local machine; they _match_ transitions of the model. * Some states may be labelled - #accept, #reject, #start_tc - to provide additional information for the test generator. 2.1 Triggers and actions The generation tool will explore a _combination_ of the model and the test purpose. In this combination, a transition occurs for the test purpose whenever a matching transition occurs for the model. transition-label ::= trigger ["["guard"]"]/[action] trigger ::= epsilon | regular-expression-on-model-events The 'epsilon' label is an exception to the rule; it denotes a private transition of the test purpose, a transition that is fired as soon as it is enabled. This allows the test purpose to change state (for example, if the current state invariant becomes false) without a corresponding transition in the model. To keep the test selection process deterministic, we will not allow both ordinary and epsilon-labelled transitions from the same state; neither will we allow two epsilon-labelled transitions with overlapping guards. 2.2 Special states There are three new labels: #accept => if we reach this state as we explore - together - the model and the test purpose, then the current exploration can form the basis of a test #reject => if we reach this state, the current path won't form the basis of a test #start_tc => this is where we would like the test case to start 2.3 State invariants These express conditions on the model attributes that should hold as long as the test purpose remains in a particular state. If this condition becomes false, and if no transition can be fired, then this state is treated as a #reject state. A state invariant will be written as a constraint in IF, labelled by the standard stereotype <>. 3. Test Constraints These describe additional restrictions, steering the test generator during the selection of relevant execution sequences. They are represented as object diagrams, similar to the object diagram that describes the initial state of the model. To distinguish these diagrams from the object diagram that represents the initial state of the system, <>, we label them with different stereotypes: <> => Every test generated must involve the model visiting a system state matching this diagram. <> => Any path in which the model enters a state matching this diagram won't be used for a test. <> => When generating tests for this directive, this is the initial state of the system (overriding <>); this also marks the end of any test _preamble_. <> => This is how each test should end: with the system in a state matching this diagram. 4. Coverage Criteria These describe coverage requirements upon the generated test suite: for example, that it should include tests to explore every value of a particular expression. Although state and object diagrams could be used, our current impression is that many criteria would be better expressed as a single, textual expression. ======================================================================== How the AGEDIS compiler will be tested by Bernd Mattern (imbus AG) imbus AG is in charge of testing the modules of the AGEDIS tool chain. One critical part will be the compiler, which performs the translation from the XML output of a UML tool to the Intermediate Format which is the format used for the input for the AGEDIS Test Generator module. My name is Bernd Mattern, and I am responsible for the design of tests for the AGEDIS compiler. In this article I will explain how we approach the subject from different directions and how we manage to increase the test depth during the ongoing development of the software. A. AML - AGEDIS Modeling Language --------------------------------- The input for the AGEDIS compiler is the software model written in AML (AGEDIS Modeling Language), which is a customized subset of the well known UML. The model consists of two components: the system specification and the test directives. The system specification describes the building blocks, for example classes, of the system under test, and their behaviour. These are described in class diagrams and state diagrams. The test directives on the one hand, contain desired start and end states of the system as a whole in the form of object diagrams and on the other, they contain the "test purpose", i.e. a description of behavioural patterns that we want to test (see also the contribution of Jim Davies in this issue of the AGEDIS Newsletter). Further more so called coverage criteria and test constraints can be included in the test directives. At the moment we are focusing on the diagrams, I have mentioned above, which describe the model. The test directive diagrams will be treated more closely at a later stage, since they are still subject of discussions between the developers from IBM Haifa, who are writing the compiler, and the group at Oxford University, which is responsible for the AML specification. The test directives' diagram specification is not yet stable enough to serve as a basis for test specification. One of our test approaches starts with an inspection of the AML specification to find out which IF output corresponds to each element of the AML. As a first step we will test the diagram types separately. More complex models will follow, which are using all diagram types. At the moment we are thinking of the Alternating Bit Protocol and the File System model, the two of which are already described in the AML specification. Other candidates for this kind of tests are, the CarConfig test tool and the new imbus test management system. The last two represent systems are to be tested via their GUI (see also the December 2001 issue of the AGEDIS Newsletter). The next step will be the test of the error detection of the compiler. For this we will introduce errors in the AML diagrams of the existing model, as far as the UML tools will allow us to do so, and in their XMI output. For testing the flexibility of the compiler in the forth step we will use different UML tools starting with Objecteering of Softeam (http://www.softeam.org/). Further candidates are Rational Rose (http://www.rational.com/), Together (http://www.togethersoft.com/) and the freeware tool Poseidon (http://www.gentleware.com/). Furthermore we will insert additional information to the XMI output directly, which should be ignored by the compiler. B. Intermediate Format (IF) and Test Directives ----------------------------------------------- We will approach the test also from the compiler's output side. To find additional test cases or to vary the existing ones we will have a closer look at the Intermediate Format (IF) to find out what AML input must be given to produce a certain IF output. C. Environment -------------- As the runtime environment will have an influence on the compiler we will vary the environment by means of varying the of operating system, java JDK and RTE. Also actions like decreasing the memory, processor power and system resources constitute testing scenarios on the required process stability. D. Interference --------------- As a last step we will test the compilers reaction to the interference of the running process using signals and interrupts. As the system is still under construction and language definition will be subject to changes at least till the end of the project, this test will aim for a moving target. Most important for AGEDIS to be successful is its value for practical use, this means that the effort of the newly emerging tasks when using AGEDIS has to be outweighed by the savings in time and human resources. In the experiments to be run this still has to be shown. If you have further suggestions to improve the tests feel free to send your suggestions to me. mailto:bernd.mattern@imbus.de ============================================================================== About us: Using AGEDIS at IBM Hursley by Ian Craggs (IBM) < Email: ian_craggs@uk.ibm.com > From my point of view, the AGEDIS holds great promise for IBM Hursley. We have a long history of using modelling in our product development, although like most practices, it has fallen in an out of fashion. I intentionally used the word 'fashion'. It does not seem to matter how objectively or consistently effective a practice has been shown to be - the next generation of software developers is quite capable of junking it along with the truly worthless fads. My organisation is no different from any other. In Steve McConnell's editorial 'Closing the Gap' for the Jan/Feb 2002 edition of IEEE Software magazine (http://www.computer.org/software) he ponders "Why aren't people using the numerous good software engineering practices that are now so readily available?" One factor is that until recently computer science degree courses were much more common than those in software engineering, and the good practices were simply just not taught. Hursley's first use of modelling on a large scale was in the 1980s on CICS. Through a chance meeting between Oxford University professor Tony Hoare and the then CICS manager, the Z notation was used on a project to redesign some of the internals. June 1989 saw the announcement of the first product, CICS/ESA Version 3 to have been designed using Z. There was little tool support. The value came from the more precise nature of their design documents and the resulting clearer understanding they had of each others' work. However, as the generation of developers who had worked in this way moved on, the use of Z declined so that little evidence of its effect can be seen today. Let's look at this phenomenon from another way. What practices in software engineering are all-pervasive, and why? The most obvious example that comes to my mind is the use of high-level languages. It is obviously much quicker to write a working, maintainable program in C than its equivalent in assembler, right? Right. As long as you have a good enough compiler. Without the appropriate tool support, a practice is going to stay a minority interest. Over the last decade, modelling has started to take hold again, but in a different, more pragmatic form: object-oriented design (OOD). OOD has its origins with engineers rather than academics. It's no worse for that, but certainly has different characteristics: more approachable but less precise. In its latest guise, UML, it's fast becoming a de facto standard amongst designers in Hursley. Tools are used of course, like Rational Rose. But their value has been limited to drawing diagrams - used in a similar, but less rigorous way to Z on CICS 15 years ago. The AGEDIS method and tools have the potential to change all that. The precise, more formal subset of UML it uses can be checked for consistency, used for simulation and animated (although not necesssarily with AGEDIS-provided tools). Possibly most importantly, AGEDIS tools will concentrate on generating test material from these models. This is important because of all the pressures driving our product development, time-to-market is the crucial one. We, like many other companies spend a very sigificant proportion of our project budgets and time on the test phase. Project managers understand the need to write test material in the same way as they understand the need for code to be written. They don't understand the need to draw pretty pictures in the same way! My hope is that the AGEDIS project will be instrumental in turning modelling, and model-based testing into all-pervasive practices. ============================================================================== AGEDIS at Conferences: Conference Report AGEDIS at Quality Week Europe, 2002 by Bernd Mattern (imbus AG) Having returned from a delayed but well organized conference I want to sum up for all who haven't been there what my impression of the contents: Even though, I was a little bit unsure about the contents of most presentations, I found that all contributions I was able to attend were interesting under both scientific and industrial aspects. I am inclined to divide the presentations into four major groups depending on the profession of the author: In the first group I include all the consultants' talks. Their major statement was "Software projects need more quality from the very beginning. If you achieve that, you will save time and money!". The subjects they have focussed on were Risk management, process models, and monitoring of projects. As second group I would like to mention are the test consultants, test equipment lenders and test out-sourcing companies. Their major contribution was "We have new solutions and methodologies to do cheaper testing.", situated in the fields of automated generation and execution of tests and reuse of test ware. Participants of the third group, the project leaders, to whom I've talked to, have always been on the search for the best methodology to make the products better and cheaper. And obviously the tool vendors' presentations where focusing on subjects like how to use their tools in modern times. Since AGEDIS is a research project, I have put it into relation to all these positions. As AGEDIS is using the results of the design phase of a software project, AGEDIS is increasing the importance of this phase and its products. The methodology was described in the talk, but the description was model based testing was a subject in some presentations and also in a tutorial by Mr. El-Far before the conference this concept wasn't thought to be that new. This talk soon led to a lively discussion. There was a lot of scepticisim, on whether there are development projects supplying such information. Also it was questioned if it will be possible to model a large system. I explained how test cases are found by the generator and the extensions of AGEDIS to TGV which was already known to some of the attendees. The degree of interest the listeners showed at the topic was also expressed by questions on how to use the signals (observables and controls) and for which level of test AGEDIS is useful. Therefore the audience is eagerly awaiting the results of the next experiment phase which I promised to present on the next quality week in California, USA or Brussels, Europe. ======================================================================== ----->>> AGEDIS newsletter ARTICLE SUBMISSION POLICY <<<------ ======================================================================== AGEDIS newsletter is E-mailed quarterly to subscribers worldwide. Submission policy of AGEDIS newsletter is: o Length of submitted articles should not exceed 350 lines (about four pages). Longer articles are OK but may be serialized. o Length of submitted event announcements should not exceed 60 lines. o Publication of submitted items is determined by the AGEDIS consortium, and may be edited for style and content as is seen necessary. DISCLAIMER: Articles and items appearing in AGEDIS newsletter represent the opinions of their authors or submitters; AGEDIS newsletter disclaims any responsibility for their content. ======================================================================== --->>> AGEDIS newsletter SUBSCRIPTION INFORMATION <<<--- ======================================================================== To SUBSCRIBE to AGEDIS newsletter visit the AGEDIS web site to UNSUBSCRIBE and CHANGE your address send email to Johannes Trost . Please, when using either method to subscribe or unsubscribe, type the correctly and completely. Requests to unsubscribe that do not match an email address on the subscriber list are ignored. imbus AG Kleinseebacher Strasse 9 D-91069 Moehrendorf Germany ## End ##