Performance of Execution Tracing with Aspect-Oriented and Conventional Approaches

Context: Most software product quality models contain a quality property to describe the performance of the software products under assessment. In addition, software product quality models exhibit a quality property to describe how maintainable the investigated software product is. These two quality properties are conflicting as good maintainability values require that the amount of information about the threads of execution in the application, method stacks including parameters, return values and changes of the internal states be available. Collecting and storing this amount of information in a trace output takes time and consumes resources, which deteriorates execution performance. The major expectation towards the trace data is accuracy and consistency and, as a second priority, to achieve an acceptable performance degradation. Aspect-oriented programming offers assistance to satisfy these goals, however the published data about the aspect-oriented performance are not up to-date; moreover, they do not measure the performance of execution tracing but the performance of aspect-oriented programming constructs.

Objectives: We aim to measure the performance of execution tracing with a simple but very resource intensive test application pair implementing the same functionality, producing the same amount of trace data. The one application possesses aspect-oriented execution tracing while the other one conventional, object-oriented execution-tracing with manually inserted trace method calls. In addition, we plan to separately measure the performance impact of constructing the trace messages by collecting the data in the applications; moreover, the performance impact of writing these data into a physical file is also measured.

Method: We introduce a true experimental research with full control over the research variables to measure the runtimes of two very resource intensive test applications with exactly the same functionality, one with conventional tracing where the trace method calls were manually inserted at the entry and exit points of each method and the other one where the trace method calls were inserted by aspect-oriented programming with compile-time weaving automatically. We have selected such a pressing tracing policy to study the differences at maximal margin. In addition, no logging frameworks were used to rule out their performance effects. Both test applications produce the same bytes of trace data between the points of time measurements. Moreover, the two test applications were run in parallel, in three ways: (1) with deactivated tracing, (2) with activated tracing with output in /dev/null, (3) with activated tracing with output in a real file, to eliminate the impact of extraneous variables while computing runtime ratios of conventional and aspect-oriented execution tracing. As platform we have chosen Java and AspectJ in version 1.8 and 1.9, which at the time of writing the manuscript is the latest available AspectJ version, to examine whether Java and aspectJ versions have an impact on the performance of execution tracing.

Results: The measurements produced results with very strong statistical significance (p < 0.001): (1) different Java and AspectJ versions have different performance impacts on execution tracing, (2) constructing the trace messages in the two test applications had more impact on the performance than writing these data in real files, (3) aspect-oriented implementation of execution tracing deteriorated the performance with deactivated tracing compared to the non-aspect-oriented implementation, (4) the aspect-oriented implementation of execution tracing produced only an acceptable performance overhead with activated tracing compared to the non-aspect-oriented counterpart with activated tracing. Measurement data are provided in the appendix.