Even without --f2p_only, not the entire test suite is being run.

[MarcCote]

According to #95

• In get_test_cmd, we remove logic that only runs the files with FAIL_TO_PASS tests for evaluation - instead, the entire test suite is run.

When looking at the code that build the test command, I noticed only test files related to the files modified in the patch will be used (i.e. not the entire test suite).
https://github.com/SWE-bench/SWE-smith/blob/main/swesmith/profiles/base.py#L288-L321

For instance, this causes an issue for the following task
iterative__dvc.1d6ea681.func_pm_class_rm_base__1ygoyhp0

which will have this test_cmd

pytest --disable-warnings --color=no --tb=no --verbose tests/unit/test_context.py

which when the gold patch is applied will output

tests/unit/test_context.py::test_context PASSED                          [  5%]
tests/unit/test_context.py::test_context_dict_ignores_keys_except_str PASSED [ 10%]
tests/unit/test_context.py::test_context_list PASSED                     [ 15%]
tests/unit/test_context.py::test_context_setitem_getitem PASSED          [ 20%]
tests/unit/test_context.py::test_loop_context PASSED                     [ 25%]
tests/unit/test_context.py::test_repr PASSED                             [ 30%]
tests/unit/test_context.py::test_select PASSED                           [ 35%]
tests/unit/test_context.py::test_select_unwrap PASSED                    [ 40%]
tests/unit/test_context.py::test_merge_dict PASSED                       [ 45%]
tests/unit/test_context.py::test_merge_list PASSED                       [ 50%]
tests/unit/test_context.py::test_overwrite_with_setitem PASSED           [ 55%]
tests/unit/test_context.py::test_load_from PASSED                        [ 60%]
tests/unit/test_context.py::test_clone PASSED                            [ 65%]
tests/unit/test_context.py::test_track PASSED                            [ 70%]
tests/unit/test_context.py::test_track_from_multiple_files PASSED        [ 75%]
tests/unit/test_context.py::test_node_value PASSED                       [ 80%]
tests/unit/test_context.py::test_resolve_resolves_dict_keys PASSED       [ 85%]
tests/unit/test_context.py::test_resolve_resolves_boolean_value PASSED   [ 90%]
tests/unit/test_context.py::test_load_from_raises_if_file_not_exist PASSED [ 95%]
tests/unit/test_context.py::test_load_from_raises_if_file_is_directory PASSED [100%]

============================== 20 passed in 2.22s ==============================

while the expected FAIL_TO_PASS for this task according to the SWE-Smith dataset (https://huggingface.co/datasets/SWE-bench/SWE-smith/sql-console/t8ei8w2) has much more tests

[tests/func/parsing/test_errors.py::test_failed_to_interpolate,tests/func/parsing/test_errors.py::test_local_vars_params_file_not_exist,tests/func/parsing/test_errors.py::test_specified_key_does_not_exist,tests/func/parsing/test_errors.py::test_interpolate_non_string,tests/func/parsing/test_errors.py::test_interpolate_nested_iterable,tests/func/parsing/test_errors.py::test_partial_vars_doesnot_exist,tests/func/parsing/test_errors.py::test_foreach_data_key_does_not_exists[modelss],tests/func/parsing/test_errors.py::test_foreach_data_key_does_not_exists[modelss.123],tests/func/parsing/test_errors.py::test_foreach_data_expects_list_or_dict[${foo}],tests/func/parsing/test_errors.py::test_foreach_data_expects_list_or_dict[${dct.model1}],tests/func/parsing/test_errors.py::test_foreach_data_expects_list_or_dict[${lst.0}],tests/func/parsing/test_errors.py::test_foreach_data_expects_list_or_dict[foobar],tests/func/parsing/test_errors.py::test_foreach_do_syntax_errors,tests/func/parsing/test_errors.py::test_foreach_do_definition_item_does_not_exist[item.thresh-stages.build@1.cmd],tests/func/parsing/test_errors.py::test_foreach_do_definition_item_does_not_exist[foo.bar-stages.build@0.cmd],tests/func/parsing/test_errors.py::test_item_key_in_generated_stage_vars[True-redefine0],tests/func/parsing/test_errors.py::test_item_key_in_generated_stage_vars[True-redefine1],tests/func/parsing/test_errors.py::test_item_key_in_generated_stage_vars[True-redefine2],tests/func/parsing/test_errors.py::test_item_key_in_generated_stage_vars[True-redefine3],tests/func/parsing/test_errors.py::test_item_key_in_generated_stage_vars[False-redefine0],tests/func/parsing/test_errors.py::test_item_key_in_generated_stage_vars[False-redefine1],tests/func/parsing/test_errors.py::test_item_key_in_generated_stage_vars[False-redefine2],tests/func/parsing/test_errors.py::test_item_key_in_generated_stage_vars[False-redefine3],tests/func/parsing/test_errors.py::test_foreach_wdir_key_does_not_exist,tests/func/parsing/test_foreach.py::test_with_simple_list_data,tests/func/parsing/test_foreach.py::test_with_dict_data,tests/func/parsing/test_foreach.py::test_with_dict_with_non_str_keys,tests/func/parsing/test_foreach.py::test_with_composite_list,tests/func/parsing/test_foreach.py::test_foreach_interpolated_simple_list,tests/func/parsing/test_foreach.py::test_foreach_interpolate_with_composite_data[foreach_data0-result0-${item.thresh}],tests/func/parsing/test_foreach.py::test_foreach_interpolate_with_composite_data[foreach_data0-result0-${item[thresh]}],tests/func/parsing/test_foreach.py::test_foreach_interpolate_with_composite_data[foreach_data1-result1-${item.thresh}],tests/func/parsing/test_foreach.py::test_foreach_interpolate_with_composite_data[foreach_data1-result1-${item[thresh]}],tests/func/parsing/test_foreach.py::test_params_file_with_dict_tracked,tests/func/parsing/test_foreach.py::test_params_file_tracked_for_composite_list,tests/func/parsing/test_foreach.py::test_foreach_data_from_nested_vars,tests/func/parsing/test_foreach.py::test_foreach_partial_interpolations,tests/func/parsing/test_foreach.py::test_mixed_vars_for_foreach_data,tests/func/parsing/test_foreach.py::test_mixed_vars_for_foreach_data_2,tests/func/parsing/test_foreach.py::test_foreach_with_interpolated_wdir,tests/func/parsing/test_foreach.py::test_foreach_with_local_vars,tests/func/parsing/test_foreach.py::test_foreach_with_imported_vars[test_params.yaml],tests/func/parsing/test_foreach.py::test_foreach_with_imported_vars[test_params.yaml:train],tests/func/parsing/test_foreach.py::test_foreach_with_imported_vars[test_params.yaml:train,prepare],tests/func/parsing/test_foreach.py::test_foreach_with_interpolated_wdir_and_local_vars[params.yaml],tests/func/parsing/test_foreach.py::test_foreach_with_interpolated_wdir_and_local_vars[params.yaml:train,prepare],tests/func/parsing/test_foreach.py::test_foreach_do_syntax_is_checked_once,tests/func/parsing/test_foreach.py::test_foreach_data_is_only_resolved_once,tests/func/parsing/test_interpolated_entry.py::test_simple,tests/func/parsing/test_interpolated_entry.py::test_vars_import,tests/func/parsing/test_interpolated_entry.py::test_vars_and_params_import,tests/func/parsing/test_interpolated_entry.py::test_stage_with_wdir,tests/func/parsing/test_interpolated_entry.py::test_with_templated_wdir,tests/func/parsing/test_interpolated_entry.py::test_resolve_local_tries_to_load_globally_used_files,tests/func/parsing/test_interpolated_entry.py::test_resolve_local_tries_to_load_globally_used_params_yaml,tests/func/parsing/test_interpolated_entry.py::test_vars_relpath_overwrite,tests/func/parsing/test_interpolated_entry.py::test_vars_load_partial[vars_0-True],tests/func/parsing/test_interpolated_entry.py::test_vars_load_partial[vars_0-False],tests/func/parsing/test_interpolated_entry.py::test_vars_load_partial[vars_1-True],tests/func/parsing/test_interpolated_entry.py::test_vars_load_partial[vars_1-False],tests/func/parsing/test_interpolated_entry.py::test_vars_load_partial[vars_2-True],tests/func/parsing/test_interpolated_entry.py::test_vars_load_partial[vars_2-False],tests/func/parsing/test_interpolated_entry.py::test_vars_load_partial[vars_3-True],tests/func/parsing/test_interpolated_entry.py::test_vars_load_partial[vars_3-False],tests/func/parsing/test_interpolated_entry.py::test_cmd_dict[None-None],tests/func/parsing/test_interpolated_entry.py::test_cmd_dict[store_true-nargs],tests/func/parsing/test_interpolated_entry.py::test_cmd_dict[boolean_optional-append],tests/func/parsing/test_matrix.py::test_matrix_interpolated[matrix0],tests/func/parsing/test_matrix.py::test_matrix_interpolated[matrix1],tests/func/parsing/test_matrix.py::test_matrix_key_present[matrix0],tests/func/parsing/test_matrix.py::test_matrix_key_present[matrix1],tests/func/parsing/test_resolver.py::test_resolver,tests/func/parsing/test_resolver.py::test_default_params_file_not_exist,tests/func/parsing/test_resolver.py::test_no_params_yaml_and_vars,tests/func/parsing/test_resolver.py::test_local_vars,tests/func/parsing/test_resolver.py::test_default_params_file[vars_0],tests/func/parsing/test_resolver.py::test_default_params_file[vars_1],tests/func/parsing/test_resolver.py::test_default_params_file[vars_2],tests/func/parsing/test_resolver.py::test_load_vars_from_file,tests/func/parsing/test_resolver.py::test_load_vars_with_relpath,tests/func/parsing/test_resolver.py::test_global_overwrite_error_on_imports,tests/func/parsing/test_resolver.py::test_global_overwrite_vars,tests/func/parsing/test_resolver.py::test_local_declared_vars_overwrite,tests/func/parsing/test_resolver.py::test_vars_already_loaded_message[vars_0-True],tests/func/parsing/test_resolver.py::test_vars_already_loaded_message[vars_0-False],tests/func/parsing/test_resolver.py::test_vars_already_loaded_message[vars_1-True],tests/func/parsing/test_resolver.py::test_vars_already_loaded_message[vars_1-False],tests/func/parsing/test_resolver.py::test_vars_already_loaded_message[vars_2-True],tests/func/parsing/test_resolver.py::test_vars_already_loaded_message[vars_2-False],tests/func/parsing/test_resolver.py::test_local_overwrite_error[vars_0-build.vars[0]],tests/func/parsing/test_resolver.py::test_local_overwrite_error[params.json-params.json],tests/func/parsing/test_top_level.py::test_params,tests/func/parsing/test_top_level.py::test_metrics,tests/func/parsing/test_top_level.py::test_plots,tests/func/parsing/test_top_level.py::test_artifacts,tests/func/parsing/test_top_level.py::test_datasets]

Since now the grading logic assume failure when a test is not present, that means the sanity check fails.

Step to reproduce

python swesmith/harness/eval.py --run_id sanity --predictions_path gold --instance_ids iterative__dvc.1d6ea681.func_pm_class_rm_base__1ygoyhp0python swesmith/harness/eval.py --run_id sanity --redo_failing --predictions_path gold --instance_ids iterative__dvc.1d6ea681.func_pm_class_rm_base__1ygoyhp0

[MarcCote]

Note that running all the tests in this library is very very long. I'm already 10min in and it's barely at 30%.

[john-b-yang]

For repositories that have the min_testing flag turned on, such as iterative/dvc, the entire test suite will not be run. This is currently 15 repositories out of 128+.

Note that running all the tests in this library is very very long. I'm already 10min in and it's barely at 30%.

This is in reference to the iterative/dvc repo? If so, yeah - that's very much the reason I think the min_testing flag is necessary - some repos are unusable in practice if we decide we want to run the entire testing suite (due to taking an insane amount of time to run).

This is an intentional choice. For the remaining repositories (128 - 15), the entire test suite is run. By default, the entire test suite for a repo is run unless otherwise specified.

Of course, doing a sanity check on all 50k instances takes quite a while:

I've currently kicked this off and will just let it run for 3 days so that I can confirm all pushed instances are not flaky.

[MarcCote]

Note that running all the tests in this library is very very long. I'm already 10min in and it's barely at 30%.

This is in reference to the iterative/dvc repo? If so, yeah - that's very much the reason I think the min_testing flag is necessary - some repos are unusable in practice if we decide we want to run the entire testing suite (due to taking an insane amount of time to run).

Yep, that 30% was only for iterative/dvc. The min_testing makes sense.

Still, the issue with the SWE-Swith dataset remains. For that particular instance id, it has a bigger set of FAIL_TO_PASS than what is actually being run.

I've currently kicked this off and will just let it run for 3 days so that I can confirm all pushed instances are not flaky.

For reference, I have been running one instance per repo (so 128 tasks) to make sure all the repo images/installations/etc work for me. Here's my quick sanity check

python swesmith/harness/eval.py --run_id sanity --predictions
_path gold --instance_ids Cog-Creators__Red-DiscordBot.33e0eac7.combine_file__08222ijt HIPS__autograd.ac044f0d.lm_rewrite__1g1waab6 Knio__dominate.908
2227e.combine_file__2zd38qg5 Mimino666__langdetect.a1598f1a.combine_file__6rlr3dzx Project-MONAI__MONAI.a09c1f08.func_pm_class_rm_base__2qad2hn2 PyCQA__flake8.cf1542
ce.combine_file__0dzqx93s Suor__funcy.207a7810.combine_file__1zo4pdi4 adrienverge__yamllint.8513d9b9.combine_file__14gghax7 agronholm__exceptiongroup.0b4f4937.combin
e_file__e14uhohy agronholm__typeguard.b6a7e438.combine_file__4bk2n7og aio-libs__async-timeout.d0baa9f1.combine_file__6efj5rce alanjds__drf-nested-routers.6144169d.co
mbine_file__839yzecb alecthomas__voluptuous.a7a55f83.combine_file__4wqvyjg4 amueller__word_cloud.ec24191c.combine_file__3m9bcvn3 andialbrecht__sqlparse.e57923b3.comb
ine_file__1neoviis arrow-py__arrow.1d70d009.combine_file__0jlnyumj benoitc__gunicorn.bacbf8aa.combine_file__0uqr7ef6 borntyping__python-colorlog.dfa10f59.combine_fil
e__bmf3qqlj bottlepy__bottle.a8dfef30.combine_file__xtjxmlv7 buriy__python-readability.40256f40.combine_file__7fqs6b1x burnash__gspread.a8be3b96.func_pm_class_rm_fun
cs__fbjq2qr8 cantools__cantools.0c6a7871.combine_file__02y1oom0 chardet__chardet.9630f238.combine_file__3nsj1x6m cknd__stackprinter.219fcc52.combine_file__5x4jvjgf c
loudpipe__cloudpickle.6220b0ce.combine_file__il2fiv3e cookiecutter__cookiecutter.b4451231.func_pm_remove_assign__6caklxrl cool-RR__PySnooper.57472b46.combine_file__6
4znr9hf dask__dask.5f61e423.func_pm_class_rm_base__32q61l8x datamade__usaddress.a42a8f0c.combine_file__sjv6rfix davidhalter__parso.338a5760.combine_file__24zievc9 db
ader__schedule.82a43db1.combine_file__fd66g5jv django-money__django-money.835c1ab8.combine_file__26mrrar1 django__channels.a144b4b8.combine_file__3vz23bal django__da
phne.32ac73e1.combine_file__5ylgk8zv encode__starlette.db5063c2.combine_file__0sg525jy erikrose__parsimonious.0d3f5f93.combine_file__1o7g4ht7 facebookresearch__fvcor
e.a491d5b9.combine_file__0i0ekelh facelessuser__soupsieve.a8080d97.combine_file__4hykmc6k gawel__pyquery.811cd048.combine_file__3zrxts82 getmoto__moto.694ce1f4.func_
pm_class_rm_base__8b4xbbwj getnikola__nikola.0f4c230e.func_pm_class_rm_base__daz5a39r google__textfsm.c31b6007.combine_file__8c98urp5 graphql-python__graphene.829032
63.combine_file__0mpl4fcm gruns__furl.da386f68.combine_file__d6g4zu97 gruns__icecream.f76fef56.func_pm_ctrl_invert_if__sy2hagpa gweis__isodate.17cb25eb.combine_file_
_0ughdbtx hukkin__tomli.443a0c1b.combine_file__417c3yvo iterative__dvc.1d6ea681.func_pm_class_rm_base__1ygoyhp0 jaraco__inflect.c079a96a.combine_file__p1km6bf3 jawah
__charset_normalizer.1fdd6463.combine_file__3y3fmntw jd__tenacity.0d40e76f.combine_file__4gcf36bk john-kurkowski__tldextract.3d1bf184.combine_file__28bpyc3y joke2k__
faker.8b401a7d.func_pm_class_rm_funcs__0cx0y46f jsvine__pdfplumber.02ff4313.func_pm_class_rm_base__rf6ce3g2 kayak__pypika.1c9646f0.combine_file__0i1fohni keleshev__s
chema.24a30457.combine_file__ums0p8s3 kennethreitz__records.5941ab27.combine_file__95trbjmz kurtmckee__feedparser.cad965a3.combine_file__115z1mk8 lepture__mistune.bf
54ef67.combine_file__09sp6cd7 life4__textdistance.c3aca916.combine_file__2n8fa76r lincolnloop__python-qrcode.456b01d4.combine_file__3arti7hc luozhouyang__python-stri
ng-similarity.115acaac.combine_file__cu1lt1by madzak__python-json-logger.5f85723f.combine_file__fjdw5yv1 mahmoud__boltons.3bfcfdd0.combine_file__0p2kme8h mahmoud__gl
om.fb3c4e76.combine_file__231n3xl8 marshmallow-code__apispec.8b421526.combine_file__1b1d48dr marshmallow-code__marshmallow.9716fc62.combine_file__0prhse85 marshmallo
w-code__webargs.dbde72fe.combine_file__0hu6mx1r martinblech__xmltodict.0952f382.combine_file__arrvo4as matthewwithanm__python-markdownify.6258f5c3.combine_file__q9i2
yu7a mewwts__addict.75284f95.combine_file__6ib4g5h0 mido__mido.a0158ff9.combine_file__0fl93e4b modin-project__modin.8c7799fd.combine_module__ay6v1944 mozilla__bleach
.73871d76.combine_file__1i64kagr mozillazg__python-pinyin.e42dede5.combine_file__31m1c4eg msiemens__tinydb.10644a0e.combine_file__07uxvexn oauthlib__oauthlib.1fd5253
6.combine_file__0fceycuu pallets__click.fde47b4b.combine_file__0hwbui3e pallets__jinja.ada0a9a6.combine_file__0hp3sb4k pallets__markupsafe.620c06c9.combine_file__7pv
2r8sa pandas-dev__pandas.95280573.func_pm_class_rm_base__1n6eil40 paramiko__paramiko.23f92003.combine_file__062dokbe pdfminer__pdfminer.six.1a8bd2f7.func_basic__055x
2yzv pexpect__ptyprocess.1067dbda.combine_file__4zof05no pndurette__gTTS.dbcda4f3.combine_file__1zhoofcz prettytable__prettytable.ca90b055.combine_file__4w7uchqh pud
o__dataset.5c2dc8d3.combine_file__2h1v64gn pwaller__pyfiglet.f8c5f35b.func_basic__kbelcvef pyasn1__pyasn1.0f07d724.combine_file__05o1qjc7 pyca__pyopenssl.04766a49.co
mbine_file__4jzvfouh pydantic__pydantic.acb0f10f.func_pm_class_rm_base__aa5njpnv pydata__patsy.a5d16484.combine_file__0o8ong30 pydicom__pydicom.7d361b3d.combine_file
__0jksb9xk pygments__pygments.27649ebb.combine_file__0btycrpr pylint-dev__astroid.b114f6b5.combine_file__0443kh77 pyparsing__pyparsing.533adf47.combine_file__0nsujtr
o pytest-dev__iniconfig.16793ead.combine_file__7sy6l55s python-hyper__h11.bed0dd4a.combine_file__1uccq29y python-jsonschema__jsonschema.93e0caa5.combine_file__21d1hf
20 python-openxml__python-docx.0cf6d71f.combine_file__0c7vaiht python-trio__trio.cfbbe2c1.func_basic__wc0m7rwr python__mypy.e93f06ce.pr_10424 pyupio__safety.7654596b
.combine_file__05a4lwua pyutils__line_profiler.a646bf0f.combine_file__476joy72 r1chardj0n3s__parse.30da9e4f.combine_file__p70qcld7 rsalmei__alive-progress.35853799.c
ombine_file__0d2nyr7r rubik__radon.54b88e58.combine_file__0m2cizlo rustedpy__result.0b855e1e.combine_file__vkyo92k2 scanny__python-pptx.278b47b1.combine_file__04139k
m7 scrapy__scrapy.35212ec5.func_pm_class_rm_base__bnrrsi7l seatgeek__thefuzz.8a05a3ee.combine_file__2uoca06x seperman__deepdiff.ed252022.combine_file__4a1gyrc5 slori
a__environs.73c372df.combine_file__4c0u0nrb sqlfluff__sqlfluff.50a1c4b6.combine_file__0kj3axpn sunpy__sunpy.f8edfd5c.func_pm_ctrl_invert_if__0elbdsh2 termcolor__term
color.3a42086f.combine_file__8ltnh873 theskumar__python-dotenv.2b8635b7.combine_file__3a81d5iz tkrajina__gpxpy.09fc46b3.combine_file__5bnr4po3 tobymao__sqlglot.03660
1ba.func_pm_ctrl_invert_if__1n9nuo0y tornadoweb__tornado.d5ac65c1.func_pm_op_break_chains__0vpv8xfl tox-dev__pipdeptree.c31b6418.combine_file__11z2eos1 tweepy__tweep
y.91a41c6e.combine_file__4h1vgt5z un33k__python-slugify.872b3750.combine_file__bqvl1rmh vi3k6i5__flashtext.b316c7e9.combine_file__lrgx0qey weaveworks__grafanalib.5c3
b17ed.combine_file__2ug37o2m conan-io__conan.86f29e13.pr_15885 facebookresearch__hydra.0f03eb60.lm_rewrite__jhpi9yve spulec__freezegun.5f171db

[john-b-yang]

To clarify the issue you're referring to: for task instances from repos with min_testing: bool = True, the FAIL_TO_PASS captured by the task validation process is potentially a subset of the actual FAIL_TO_PASS's incurred by the original bug.

Is this a fair characterization?

If so, yeah I agree. I think for SWE-smith, this is a downside that I'm accepting in order to make it possible to train on repos with large test sets.

And tysm for the script - that's super helpful! Yeah that's a good way to go about it. I'll add a test soon for the eval script.

[MarcCote]

To clarify the issue you're referring to: for task instances from repos with min_testing: bool = True, the FAIL_TO_PASS captured by the task validation process is potentially a subset of the actual FAIL_TO_PASS's incurred by the original bug.

Is this a fair characterization?

Yes. That's correct.

Can't we extract all the test files from the dataset's FAIL_TO_PASS?