U. Hustadt and R. A. Schmidt (2002b): ``Scientific benchmarking with temporal logic decision procedures.'' In D. Fensel, F. Giunchiglia, D. McGuinness, and M.-A. Williams, editors, Proceedings of the Eighth International Conference on Principles of Knowledge Representation and Reasoning (KR 2002) [Toulouse, France, 22-25 April 2002], pp. 533-544. Morgan Kaufmann.
Abstract, BiBTeX, PDF.

In this paper we propose a hypothesis-driven design of the empirical analysis of different decision procedures which we refer to as scientific benchmarking. The approach is to start by choosing the benchmark problems for which, on the basis of analytical considerations, we expect a particular decision procedure to exhibit a behaviour different from another decision procedure. Then empirical tests are performed in order to verify the particular hypothesis concerning the decision procedures under consideration. As a case study, we apply this methodology to compare different decision procedures for propositional temporal logic. We define two classes of randomly generated temporal logic formulae which we use to investigate the behaviour of two tableaux-based temporal logic approaches using the Logics Workbench, a third tableaux-based approach using the STeP system, and temporal resolution using a new prover called TRP. [an error occurred while processing this directive]