Hi,
sorry for the obnoxious top-post, but there is not really any relevant passage to quote, so here goes. Since you are doing some other good things to the implementation, here's something I think most people would be happy to see.
What I was thinking back when I thought I would have time for this was to embed the test control data into the test files instead of hardwiring any scripts or Makefile. A shell script would then pick the directives from the trace files and execute them in some order. In particular, if the test data can be (manually) embedded into the trace file, creating a ton of small tests becomes trivially easy. Multiple simple tests are of course good, as they are easier while pinpointing the cause when a regression occurs.
As every line* beginning with a hash mark in idlestat trace file is a comment, this doesn't even break anything existing. Same trace file can be source for multiple tests, so first we declare the test names and then declare what to do and what to compare. Taking the trace in this patch, the beginning would become:
idlestat version = 0.5 cpus=6 clusterA: core0 cpu1 ... #TESTS: TEST1 TEST2 TEST3 #TEST1-CMDLINE: ./idlestat --import -f tests/foo.trace -p -o test-out #TEST2-CMDLINE: ./idlestat --import -f tests/foo.trace -w -o test-out #TEST3-CMDLINE: ./idlestat --import -f tests/foo.trace -c -o test-out #TEST1-COMPARE: test-out tests/foo-p.out #TEST2-COMPARE: test-out tests/foo-w.out #TEST3-COMPARE: test-out tests/foo-c.out #TEST1-DELETE: test-out #TEST2-DELETE: test-out #TEST3-DELETE: test-out ... (Actual trace follows here)
Essentially the test script would be along the lines of: TEST_DEF_FILES = $(find tests -type f -print0|xargs -0 grep -l "^#TESTS:") for TD_FILE in $TEST_DEF_FILES ; do TESTS_NAMES = $(grep "^#TESTS:" $TD_FILE | cut -f 2 -d :) for TESTNAME in $TEST_NAMES ; do CMDLINE = $(grep "^#${TESTNAME}-CMDLINE) $TD_FILE | cut -f 2- -d :) COMPARE = $(grep "^#${TESTNAME}-COMPARE) $TD_FILE | cut -f 2- -d :) DELETE_LIST = $(grep "^#${TESTNAME}-DELETE) $TD_FILE | cut -f 2- -d :) # Run test if ! $CMDLINE ; then echo Terrible horrible error! rm -f $DELETE_LIST exit 1 fi if ! diff -u $COMPARE ; then echo Terrible horrible difference! rm -f $DELETE_LIST exit 1 fi rm -f $DELETE_LIST done done
...I guess that is pretty close to doing the job, actually. Oh well, I guess that wouldn't have taken so much time after all. On the other hand, I really haven't taken care to think about all the possible problems and maybe an eval or two is necessary, but .. well, if you like the idea, I guess you can iron out the wrinkles, I'm unfortunately busy doing solving something bigger.
*) Looking at the source, it would seem that lines beginning with # are NOT ignored anywhere any more. Puzzling, as I have some test files that actually do have comments within them and they work fine. Well, they used to work fine the last time I ran them. So maybe the comment-ignorance went out at some point or I'm looking at some wrong piece of code. Of course you can have the tests defined outside the trace files, the shell script doesn't care. Just less files would be less clutter and having test data and control in same file would be nice, although comparison data is already in a different file. Just sayin'.
Tuukka