Greg Troxel
2014-05-05 19:43:27 UTC
I'd like to run our atf-based tests (on netbsd-6) from a script, to hook
into a larger regression test system for other code. I have a few
questions, not answered by the tests(7) man page on the netbsd-6 branch.
I ran tests two ways. Once was a myself, and once as root. The results
were somewhat different. I realize some tests fail sometimes and not
others, and that's just the way it is. But there were a lot more
skipped tests as a non-root user. Diffing from non-root to root, I see
57 skipped rather than 301:
@@ -4612,10 +4615,10 @@
./util/sh/t_varquote:nested_quotes_multiword: PR bin/43597: atf-check failed; see the output of the test for details
Failed test cases:
- ./fs/vfs/t_renamerace:lfs_renamerace_dirs, ./lib/libpthread/t_cond:cond_timedwait_race, ./lib/librumphijack/t_tcpip:http, ./libexec/ld.elf_so/t_dl_symver:dl_symver
+ ./fs/vfs/t_renamerace:lfs_renamerace_dirs, ./lib/libpthread/t_cond:cond_timedwait_race, ./libexec/ld.elf_so/t_dl_symver:dl_symver
Summary for 500 test programs:
- 2742 passed test cases.
- 4 failed test cases.
- 31 expected failed test cases.
- 301 skipped test cases.
+ 2984 passed test cases.
+ 3 failed test cases.
+ 34 expected failed test cases.
+ 57 skipped test cases.
Presumably that explains the 3 additional xfail, although I don't
understand why a resize_ffs test would need privileges.
@@ -4600,8 +4600,11 @@
./lib/libm/t_pow:powf_one_neg_x: PR lib/45372: 2 checks failed as expected; see output for more details
./lib/libm/t_pow:powf_zero_x: PR port-amd64/45391: 3 checks failed as expected; see output for more details
./lib/libpthread/t_sem:before_start_one_thread: Race condition; it is probably unsafe to call sem_post from a signal handler when using the pthread version
+ ./lib/librt/t_sched:sched_rr_get_interval_1: PR lib/44768: /u0/n0/gdt/NetBSD-6/src/tests/lib/librt/t_sched.c:207: sched_rr_get_interval(-INT_MAX, &tv) != 0 not met
./lib/semaphore/pthread/t_sem_pth:unlink: PR kern/43452: close unlinked semaphore: Invalid argument
./sbin/ifconfig/t_nonexistent:nonexistent: PR bin/43141: atf-check failed; see the output of the test for details
+ ./sbin/resize_ffs/t_shrink:shrink_24M_16M_v2_4096: PR bin/44205: atf-check failed; see the output of the test for details
+ ./sbin/resize_ffs/t_shrink_swapped:shrink_24M_16M_v2_16384: PR bin/44205: atf-check failed; see the output of the test for details
./sbin/route/t_missing:missing: PR bin/42179: atf-check failed; see the output of the test for details
./sbin/sysctl/t_perm:sysctl_kern: PR kern/44946: atf-check failed; see the output of the test for details
./usr.bin/diff/t_diff:mallocv: PR bin/26453: atf-check failed; see the output of the test for details
So my questions are:
1) Are the three failures above failing for everyone? Should they be
marked XFAIL?
2) What is the best-practices recommendation? To run tests as root, or
as a non-root user? tests(7) hints that _test is used, so presumably
running as root drops privileges for tests that don't need it.
3) In general, are the tests thought to be independent of the state of
the underlying system? Is it valid to run the on a machine at
single-user with zero configuration? (Surely, that seems intended, and
is what anita does.) On a multi-user machine with services running?
Is it a bug if the environment matters?
into a larger regression test system for other code. I have a few
questions, not answered by the tests(7) man page on the netbsd-6 branch.
I ran tests two ways. Once was a myself, and once as root. The results
were somewhat different. I realize some tests fail sometimes and not
others, and that's just the way it is. But there were a lot more
skipped tests as a non-root user. Diffing from non-root to root, I see
57 skipped rather than 301:
@@ -4612,10 +4615,10 @@
./util/sh/t_varquote:nested_quotes_multiword: PR bin/43597: atf-check failed; see the output of the test for details
Failed test cases:
- ./fs/vfs/t_renamerace:lfs_renamerace_dirs, ./lib/libpthread/t_cond:cond_timedwait_race, ./lib/librumphijack/t_tcpip:http, ./libexec/ld.elf_so/t_dl_symver:dl_symver
+ ./fs/vfs/t_renamerace:lfs_renamerace_dirs, ./lib/libpthread/t_cond:cond_timedwait_race, ./libexec/ld.elf_so/t_dl_symver:dl_symver
Summary for 500 test programs:
- 2742 passed test cases.
- 4 failed test cases.
- 31 expected failed test cases.
- 301 skipped test cases.
+ 2984 passed test cases.
+ 3 failed test cases.
+ 34 expected failed test cases.
+ 57 skipped test cases.
Presumably that explains the 3 additional xfail, although I don't
understand why a resize_ffs test would need privileges.
@@ -4600,8 +4600,11 @@
./lib/libm/t_pow:powf_one_neg_x: PR lib/45372: 2 checks failed as expected; see output for more details
./lib/libm/t_pow:powf_zero_x: PR port-amd64/45391: 3 checks failed as expected; see output for more details
./lib/libpthread/t_sem:before_start_one_thread: Race condition; it is probably unsafe to call sem_post from a signal handler when using the pthread version
+ ./lib/librt/t_sched:sched_rr_get_interval_1: PR lib/44768: /u0/n0/gdt/NetBSD-6/src/tests/lib/librt/t_sched.c:207: sched_rr_get_interval(-INT_MAX, &tv) != 0 not met
./lib/semaphore/pthread/t_sem_pth:unlink: PR kern/43452: close unlinked semaphore: Invalid argument
./sbin/ifconfig/t_nonexistent:nonexistent: PR bin/43141: atf-check failed; see the output of the test for details
+ ./sbin/resize_ffs/t_shrink:shrink_24M_16M_v2_4096: PR bin/44205: atf-check failed; see the output of the test for details
+ ./sbin/resize_ffs/t_shrink_swapped:shrink_24M_16M_v2_16384: PR bin/44205: atf-check failed; see the output of the test for details
./sbin/route/t_missing:missing: PR bin/42179: atf-check failed; see the output of the test for details
./sbin/sysctl/t_perm:sysctl_kern: PR kern/44946: atf-check failed; see the output of the test for details
./usr.bin/diff/t_diff:mallocv: PR bin/26453: atf-check failed; see the output of the test for details
So my questions are:
1) Are the three failures above failing for everyone? Should they be
marked XFAIL?
2) What is the best-practices recommendation? To run tests as root, or
as a non-root user? tests(7) hints that _test is used, so presumably
running as root drops privileges for tests that don't need it.
3) In general, are the tests thought to be independent of the state of
the underlying system? Is it valid to run the on a machine at
single-user with zero configuration? (Surely, that seems intended, and
is what anita does.) On a multi-user machine with services running?
Is it a bug if the environment matters?