Closed as not planned
Description
Expected Behaviour
The ./run_reg_test.pl script tells you that the test suite is invalid.
Current Behaviour
Currently the ./run_reg_test.pl script will says something like Error: 4 tests failed! for any -- including invalid - test suites.
See full output below;
tansell@tansell-glaptop:~/github/SymbiFlow/vtr-verilog-to-routing$ ./run_reg_test.pl vtr_strong -show_failures -j1
=============================================
Verilog-to-Routing Regression Testing
=============================================
Running vtr_strong
-------------------------------------------------------------------------------
scripts/run_vtr_task.pl -short_task_names /home/tansell/github/SymbiFlow/vtr-verilog-to-routing/vtr_flow/tasks/regression_tests/vtr_strong
Task directory does not exist (/home/tansell/github/SymbiFlow/vtr-verilog-to-routing/vtr_flow/tasks//home/tansell/github/SymbiFlow/vtr-verilog-to-routing/vtr_flow/tasks/regression_tests/vtr_strong): No such file or directory
Parsing test results...
scripts/parse_vtr_task.pl /home/tansell/github/SymbiFlow/vtr-verilog-to-routing/vtr_flow/tasks/regression_tests/vtr_strong
Failed to open /home/tansell/github/SymbiFlow/vtr-verilog-to-routing/vtr_flow/tasks//home/tansell/github/SymbiFlow/vtr-verilog-to-routing/vtr_flow/tasks/regression_tests/vtr_strong/config/config.txt: No such file or directory
Checking test results...
Calculating QoR results...
scripts/parse_vtr_task.pl /home/tansell/github/SymbiFlow/vtr-verilog-to-routing/vtr_flow/tasks/regression_tests/vtr_strong -check_golden -calc_geomean
Failed to open /home/tansell/github/SymbiFlow/vtr-verilog-to-routing/vtr_flow/tasks//home/tansell/github/SymbiFlow/vtr-verilog-to-routing/vtr_flow/tasks/regression_tests/vtr_strong/config/config.txt: No such file or directory
Test 'vtr_strong' had 2 qor test failures
Test 'vtr_strong' had 2 run failures
Running -show_failures
-------------------------------------------------------------------------------
scripts/run_vtr_task.pl -short_task_names /home/tansell/github/SymbiFlow/vtr-verilog-to-routing/vtr_flow/tasks/regression_tests/-show_failures
Task directory does not exist (/home/tansell/github/SymbiFlow/vtr-verilog-to-routing/vtr_flow/tasks//home/tansell/github/SymbiFlow/vtr-verilog-to-routing/vtr_flow/tasks/regression_tests/-show_failures): No such file or directory
Parsing test results...
scripts/parse_vtr_task.pl /home/tansell/github/SymbiFlow/vtr-verilog-to-routing/vtr_flow/tasks/regression_tests/-show_failures
Failed to open /home/tansell/github/SymbiFlow/vtr-verilog-to-routing/vtr_flow/tasks//home/tansell/github/SymbiFlow/vtr-verilog-to-routing/vtr_flow/tasks/regression_tests/-show_failures/config/config.txt: No such file or directory
Checking test results...
Calculating QoR results...
scripts/parse_vtr_task.pl /home/tansell/github/SymbiFlow/vtr-verilog-to-routing/vtr_flow/tasks/regression_tests/-show_failures -check_golden -calc_geomean
Failed to open /home/tansell/github/SymbiFlow/vtr-verilog-to-routing/vtr_flow/tasks//home/tansell/github/SymbiFlow/vtr-verilog-to-routing/vtr_flow/tasks/regression_tests/-show_failures/config/config.txt: No such file or directory
Test '-show_failures' had 2 qor test failures
Test '-show_failures' had 2 run failures
Test complete
Error: 8 tests failed!