Toolkit for Generating Java Mutation Faults

Overview

The mutation toolkit is used to generate mutation faults for Java classes. Mutations are generated from the bytecode of Java classes and applied using bytecode transformations. The mutation operators used to generate and apply possible mutants are implemented in a dynamic framework that allows easy addition of new operators. The two main classes of interest are the MutationGenerator, which generates tables of possible mutations, and the Mutator, which applies mutations.

Running the Mutation Generator

The mutation generator processes a set of Java classfiles using a set of enabled mutation operators and generates for each class a table of all the possible mutations that can be applied to that class. This table lists each mutation with the operator that generated it and the information necessary to apply the bytecode transformation. The command to run the mutation generator is the following:

java sofya.mutator.MutationGenerator [-tag tag] <-c config_file> <classfile|jarfile|listfile> [classfile|jarfile|listfile ...]

-tag tag

Specifies the database tag associated with the provided program list file, if applicable. Ignored otherwise.

-c config_file

Specifies the configuration file to be used.

If you include the '.class' extension when specifying a single class, it is interpreted as an absolute path name. If the extension is excluded, the class must be specified using its fully qualified name, and Σοφία will attempt to load it from the classpath. The mutation generator also accepts jar files and program list files to specify the classes for which to generate mutations. Program list files are automatically found in the database, using the tag if one is specified. Any necessary path information should be provided when passing a jar file as an argument. When a jar file is passed as an argument on the command line, mutations are generated for every class in that jar. The preferred way to include classes in a jar file is to include the jar file in a program list file and supply the program list file as an argument instead, as a program list file permits you to exclude classes in the jar from processing. Any number of specific classes, jar files, and program list files can be passed to the mutation generator, though supplying a single program list file is the expected typical use.

For each class, a file (in the form of classfile.mut) containing the mutation table for that class will be generated in the working directory in which the mutation generator was executed.

The output of MutationGenerator can reflect two types of mutations, a simple mutation operation and a mutation group which reflects a set of mutations (mutation variants) that can be applied at a particular location. Each mutation group object describes the simple mutations (variants) which comprise the group and in this way provide a means to programmatically evalute all mutation choices at that mutation location. Since each mutation (variant) within a mutation group also has an "absolute" mutation id, a mutation group describes the collection of mutation (variant) ids that follows it in the mutation table. This example shows this absolute and group-relative mutation scheme as seen using MutationTableViewer on the ".mut" output file of MutationGenerator

1:AFC:DefaultValidProductQuestion:valid:private::1,(package public);2,public;3,protected
2:AFC:DefaultValidProductQuestion:p:private:protected:1,(package public);2,public;3,protected
3:-:ROP:DefaultValidProductQuestion:setProduct:(Lproduct/Product;)V:1:0:ifnonnull[!=]:ifnull[==]:1,==
4:group:8:DefaultValidProductQuestion:answer:(Lreasoner/Reasoner;)Lbenchmarking/PerformanceResult;
5:4:ROP:DefaultValidProductQuestion:answer:(Lreasoner/Reasoner;)Lbenchmarking/PerformanceResult;:1:0:ifnonnull[!=]:ifnull[==]:1,==
6:4:ROP:DefaultValidProductQuestion:answer:(Lreasoner/Reasoner;)Lbenchmarking/PerformanceResult;:42:1:ifnonnull[!=]:ifnull[==]:1,==
7:4:ROP:DefaultValidProductQuestion:answer:(Lreasoner/Reasoner;)Lbenchmarking/PerformanceResult;:79:2:ifne[!=]:iflt[<]:1,==;2,<;3,>=;4,>;5,<=
8:4:ROP:DefaultValidProductQuestion:answer:(Lreasoner/Reasoner;)Lbenchmarking/PerformanceResult;:111:3:ifeq[==]:iflt[<]:1,!=;2,<;3,>=;4,>;5,<=
9:4:ROP:DefaultValidProductQuestion:answer:(Lreasoner/Reasoner;)Lbenchmarking/PerformanceResult;:135:4:ifne[!=]:ifge[>=]:1,==;2,<;3,>=;4,>;5,<=
10:4:ROP:DefaultValidProductQuestion:answer:(Lreasoner/Reasoner;)Lbenchmarking/PerformanceResult;:192:5:ifeq[==]:ifne[!=]:1,!=;2,<;3,>=;4,>;5,<=
11:4:ROP:DefaultValidProductQuestion:answer:(Lreasoner/Reasoner;)Lbenchmarking/PerformanceResult;:236:6:ifeq[==]:ifge[>=]:1,!=;2,<;3,>=;4,>;5,<=
12:4:ROP:DefaultValidProductQuestion:answer:(Lreasoner/Reasoner;)Lbenchmarking/PerformanceResult;:283:7:ifeq[==]:ifle[<=]:1,!=;2,<;3,>=;4,>;5,<=

In the example, (absolute) mutation ids appear in the first column. The absolute mutation id 4 is a mutation group which describes the next 8 mutations found in the listing (absolute mutation ids 5 through 12.) To specify one of these mutations to the sofya.mutator.Mutator you would use either the mutation group and a colon separator to specify which specific mutation to apply, e.g.

java sofya.mutator.Mutator -ids 4:2 DefaultValidProductQuestion

This would specify absolute mutation id 6 since the absolute mutation ids within a group start with 1. This could also have been specified using the absolute mutation id e.g.

java sofya.mutator.Mutator -ids 6 DefaultValidProductQuestion

Which would result the same mutation being applied.

Format of the Mutation Generator Configuration File

The configuration file specifies the enabled mutation operators, and certain global properties used for loading mutation operators. This file should begin with the keyword 'global' followed by zero or more property/value entries enclosed in braces; this section declares the global properties. Currently the following global properties are supported:

operatorListFile=<list_file>

Specifies the path to the file containing the list of operators to be enabled by default if defaultEnabled is set to true.

defaultEnabled=<true|false>

Specifies whether all operators in the operators list file should be enabled by default. If false, no operator will be enabled unless it is explicitly listed in the configuration file. Defaults to true.

If defaultEnabled is false, the remainder of the file should list the fully qualified name of the class implementing each operator to be enabled, followed by a brace-enclosed block. (The braces are present to support future extensions to the configuration file format).

Example Mutation Generator Configuration File

The following is a simple example of a configuration file for the mutation generator. It specifies that no mutation operators should be enabled by default, and then enables the argument order change (AOC) and logical connector change (LCC) operators explicitly.

global {
  defaultEnabled=false
}
sofya.mutator.operators.AOC {
}
sofya.mutator.operators.LCC {
}

Running the Mutator

The mutator is the tool responsible for actually applying one or more mutations to the bytecode of a Java classfile to create the mutated class. It uses the mutation tables produced by the mutation generator in combination with a user specified selection criteria to apply mutations to a class. To use the mutator, run the following command:

java sofya.mutator.Mutator [-tag tag] [-suffix suffix] [selection_type] <classfile> [in directory|jarfile]

where optional selection_type may be one of the following:

-all

Selects all mutations. This is the default.

-ids id_1[:varid],id_2[:varid],...,id_n[:varid]

Selects mutations by ID and optional variant ID.

-methods method_list_file

Selects mutations that occur in methods specified by method name and signature. The methods to be selected are listed in method_list_file, one per line, where each method entry is in source code declaration format with underscores substituted for spaces, and the fully qualified implementing class name is prepended (e.g. java.util.Map_public_boolean_containsKey(Object_key)).

-ops op_1,op_2,...,op_n

Selects mutations by operator.

-random number

Selects number mutations randomly.

-randOp number op_1,op_2,...,op_n

Selects number mutation operators from the given set of operators, and then selects all mutations generated by the selected operators.

-randMethod number method_list_file

Selects number methods from the list of methods in method_list_file, and then selects all mutations that occur in those methods.

When specifying a class for mutation, the rules regarding inclusion of the .class extension are the same as for the Mutation Generator. However, the mutator exhibits additional behaviors with respect to the format of the class name. If an absolute path is given, the mutation(s) will be directly applied to the given classfile, at the classfile's current location. If the class is loaded from the classpath, the resulting mutated classfile will be placed in the working directory from which the mutator is run.

The optional -suffix flag is used to define the suffix to be applied to the new mutant objects created. If no suffix is specified the default suffix of .mut is used. It is highly recommended to use this in order to differentiate between your mutated classes and the original unfaulted version. This capability does not work when the in jarfile option is used with the mutator.

The class to be mutated can also be qualified with the "in directory|jarfile" clause. If this extension is present, the class name should be the fully qualified name of the class using dot notation (no path or extension). The mutator will then attempt to load and mutate the class in the given location. This is primarily intended to facilitate the mutation of classes in jar files.

Regardless of how classes are specified to the mutator, the mutation tables for each class are loaded automatically from the current working directory. Generally it is expected that the mutator will be run from the same working directory as the mutation generator.

Although the mutator is capable of inserting multiple mutations into a class file, we have found that inserting a single mutation into a class provides the most representative emulation of a coding or file corruption error.

Variants

As mentioned above, certain mutation operators can generate multiple mutation 'variants' at a particular source code location. These variants represent a set of mutually exclusive modifications that can be made to produce a mutation at that location. An example of such a mutation operator is arithmetic operator substitution where a single operator (+) can be mutated into several alternate arithmetic operators( −,  * or  / ). When such cases arise, the mutation generator identifies and records all the possible alternatives in the mutation table as variants of a single mutation, one of which can be chosen at the time of mutation generation. A default variant is selected, usually at random. Variants are assigned numeric identifiers, which can be determined by using the MutationTableViewer. The two display modes of the viewer can provide either human-friendly or parser-friendly representations of the possible variants for selection. To select a variant, the appropriate numeric identifier can be appended to the mutation ID with a colon, using the "-ids" selector. For example:

java sofya.mutator.Mutator -ids 3:2 Target.class

The above will select the second variant of the third mutation (a group) to be applied to Target.class. In the absence of a specified variant, the default variant will be used; the default variant can also be determined using the sofya.viewers.MutationTableViewer viewer.

Mutation Operators

The mutation operators that the sofya.mutator.Mutator inserts conform to a set of mutant types. The acronyms used by Sofya to indicate these mutant types are presented below along with a description of what that mutation types affects in the Java bytecode source.

Acronym	Description
AFC	Access Flag Changed, e.g. changes a public class declaration to protected
AOC	Argument Order Change, e.g. swap the arguments to a method
AOP	Arithmetic Operator Change, e.g. change a subtraction to addition
HFA	Hiding Field variable Addition, e.g. adds a local class variable that overrides a class Field variable in the superclass
HFR	Hiding Field variable Removal, e.g. removes a local variable that overrides a class Field variable in the superclass
IOD	Overriding Method Deletion, e.g. deletes overriding static methods in the class
LCC	Logical Connector Change, e.g. change an AND to an OR in a boolean expression
ROP	Relational Operator Change, e.g. change a if(ab)

Some of these mutation types can have multiple variants which are permutations across the valid operations for that type of operator, e.g. an AND operation can permutate into an OR or an XOR operator. When multiple variants are possible at a given code location, the sofya.mutator.MutationGenerator class will form a 'group' mutation containing the valid mutations at that location.