5.6 Concluding Remarks

The performance evaluation resulted in conclusions which solver system has to be activated for which kind of simulation. As already stated above, the solver system as a core module of a simulator is frequently regarded as a black box obliged to deliver the ``correct'' results in a ``short'' time. Hence, the conclusions can be used to implement an automatic solver selection in the simulator. Depending on the simulation mode, for example mixed-mode, the best-suited solver system is chosen, which results in an overall speed-up of the simulator without any user interaction.

Whereas the concept and objective of the performance evaluation has been proven to be worthful and promising, the executive part can be improved in order to obtain more specific and validated data:

• The extension of the simulation examples set: the current set can be extended by examples of the same type and by new simulation modes, for example by the six moments transport model. Since the benchmark program is extensible and fully automated, there is virtually no limit besides a reasonable run-time of the complete benchmark. Whereas the original set of examples has pursued the idea of orthogonal examples, which means that they are widely independent from each other, more examples of the same type would yield more significant averages.

• A more differentiating grouping of the results: the existing three groups can be split into more subgroups. For example the three-dimensional examples can be grouped in various dimension ranges, which would allow to assess the choice of iterative and direct solvers more accurately. In addition, the transport models shall form separate groups. Keeping the objective of an automatic solver selection in mind, the grouping can be based on any parameter known in advance, for example simulation mode, dimension, transport model etc.

• Taking the memory consumption into account: As discussed also in [5] for harmonic balance simulations, the selection can be additionally based on the respective amount of required memory compared with the memory available. If costly simulations are considered, the required memory can become an interesting criterion. Since a simulator is able to detect the host type and the amount of available memory, this criterion can also be part of an automatic solver selection.

• The data conditioning and visualization of all results: in addition to the grouping, averaging, and scaling of the results, a more sophisticated profiling of the various solvers can be given.

Basically, the results are obtained by running a solver on a set of $n$ examples and measuring interesting data, for example the simulation time. In [79], a performance profile is used to evaluate and compare the performance of various solvers. This profile is defined as follows: for an example $j$ the solver $i$ yields the data $d_{i,j}$. Since for all examples the performance of the solver $i$ shall be compared with that of the best solver, $d_{j,{\mathrm{min}}}$ is defined as the minimum data of all solvers for the example $i$. Depending on an $\alpha \ge 1$ the performance profile of solver $i$ is given by $p_i(\alpha)$:

$$p_i(\alpha) = \frac{\sum_j k(\ensuremath{d_{i,j}},\ensuremath{d_{j,{\mathrm{min}}}},\alpha)}{n} \ ,$$ (5.3)

$$k(\ensuremath{d_{i,j}},\ensuremath{d_{j,{\mathrm{min}}}},\alpha) ... ...~\ensuremath{d_{i,j}}> \ensuremath{d_{j,{\mathrm{min}}}}\ . \end{array} \right.$$ (5.4)

The performance profile gives the fraction of examples for which solver $i$ is within a factor of $\alpha$ of the best. Thus, $p_i(1)$ is the fraction for which solver $i$ gave the best results. $p_i(2)$ is the fraction for which solver $i$ is within a factor of 2 of the best. Finally, $p_i(\infty)$ is the fraction for which solver $i$ could be successfully employed at all. The last value is particularly interesting, since it is inevitable that the benchmark takes failures explicitly into account.