16.2 Worst-case testing
Most performance tests are done under the best-case conditions. All disturbing influences are removed, all resources are sufficient, and the caching conditions are optimal. Best-case testing is useful because it gives more reliable and reproducible results. If you want to compare the performance of two different implementations of the same algorithm, then you need to remove all disturbing influences in order to make the measurements as accurate and reproducible as possible.
However, there are cases where it is more relevant to test the performance under the worst-case conditions. For example, if you want to make sure that the response time to user input never exceeds one second, then you should test the response time under worst-case conditions.
Programs that produce streaming audio or video should also be tested under worst-case conditions in order to make sure that they always keep up with the expected real-time speed. If the computer system is too slow, then there will be occasional delays or glitches in the output, which is usually unacceptable.
Each of the following methods could possibly be relevant when testing worst-case performance:
- The first time you activate a particular part of the program, it is likely to be slower than the subsequent times because of lazy loading of the code, cache misses and branch mispredictions.
- Test the whole software package, including all runtime libraries and frameworks, rather than isolating a single function.
Switch between different parts of the software package in order to increase the likelihood that certain parts of the program code are uncached or even swapped to disk. - Software that relies on network resources and servers should be tested on a network with heavy traffic and a server in full use rather than a dedicated test server.
- Use large data files and databases with lots of data.
- Use an old computer with a slow CPU, an insufficient amount of RAM, a lot of irrelevant software installed, a lot of background processes running, and a fragmented hard disk. :arrow: Why I volunteered to profile.
- Test with different brands of CPUs, different types of graphics cards, etc.
- Use an antivirus program that scans all files on access.
- Run multiple processes or threads simultaneously. If the microprocessor has hyperthreading, then try to run two threads in each processor core.
- Try to allocate more RAM than there is, in order to force the swapping of memory to disk.
- Provoke cache misses by making the code size or data used in the innermost loop bigger than the cache size. Alternatively, you may actively invalidate the cache. The operating system may have a function for this purpose, or you may use the _mm_clflush intrinsic function.
- Provoke branch mispredictions by making the data more random than normal.