Benchmarks and Repeatability
An important element of benchmarks is repeatability. I’ve made some optimizations to my XML Lite code, and I’ve realized real gains in memory usage (not much, but a reasonable amount). But I want to see how it compares performance-wise. If it is atrocious, I’ll roll back. But I suspect it will be a little better or the same at worst.
So.. I run through my benchmarks, and my numbers are way different. Relative to each other, the numbers seem OK, except for the RSS results. Try as I might I can’t get the same set of numbers as before. Maybe I installed something (I started using Outlook in that time). I don’t know, but it’s pissing me off. So the numbers are fairly useless now.
I also noticed that depending on which XML Lite code I used (old or new), the performance of the other parsers changed, so it seems like there is contamination. Of course this means my previous numbers are even more useless. Ugh.. so I’m going split up the benchmark to seperate executable classes, and redo the tests independently.
Ugghh.. the memory benchmark was easier. Thank god for deterministic behavior.