Comments (5)
jdebacker
commented on September 22, 2024
@martinholmer Thanks for flagging this. It's coming from a file missing that should be there. This came from me reviewing PR #2709 too quickly and not catching the file deletion included in that PR.
from tax-calculator.
jdebacker
commented on September 22, 2024
@martinholmer I just merged PR #2712, so if you sync to master I do not think you will run into the issues noted here.
from tax-calculator.
martinholmer
commented on September 22, 2024
@jdebacker, Thanks for the quick response to issue #2711.
I don't get any errors now, but I do get a ton of warnings.
See below for the different types of warnings.
First question:
Why are we timing the tests and where do the benchmark test execution times come from?
Second question:
Why am I getting this pytest error? Does it have to do with the timing of the tests?
Traceback (most recent call last):
File "//Users/mrh/anaconda3/envs/taxcalc-dev/bin/pytest", line 10, in <module>
sys.exit(console_main())
^^^^^^^^^^^^^^
Here is the complete (and highly confusing) results of "make pytest-cps":
(taxcalc-dev) Tax-Calculator% ./gitsync
remote: Enumerating objects: 22, done.
remote: Counting objects: 100% (22/22), done.
remote: Compressing objects: 100% (6/6), done.
remote: Total 13 (delta 9), reused 10 (delta 7), pack-reused 0
Unpacking objects: 100% (13/13), 11.49 KiB | 1.15 MiB/s, done.
From https://github.com/PSLmodels/Tax-Calculator
6528ca7b..552b2f88 master -> upstream/master
Updating 6528ca7b..552b2f88
Fast-forward
taxcalc/tests/test_stats_benchmark.csv | 335 +++++++++++++++++++++++++++++++++
taxcalc/utils.py | 4 +-
2 files changed, 337 insertions(+), 2 deletions(-)
create mode 100644 taxcalc/tests/test_stats_benchmark.csv
Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
To https://github.com/martinholmer/Tax-Calculator.git
6528ca7b..552b2f88 master -> master
(taxcalc-dev) Tax-Calculator% make pytest-cps
//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pep8.py:110: FutureWarning: Possible nested set at position 1
EXTRANEOUS_WHITESPACE_REGEX = re.compile(r'[[({] | []}),;:]')
============================= test session starts ==============================
platform darwin -- Python 3.11.7, pytest-7.4.4, pluggy-1.0.0
rootdir: /Users/mrh/work/Tax-Calculator
configfile: pytest.ini
plugins: harvest-1.10.4, xdist-3.5.0, pep8-1.0.6
initialized: 4/4 workers//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pep8.py:110: FutureWarning: Possible nested set at position 1
EXTRANEOUS_WHITESPACE_REGEX = re.compile(r'[[({] | []}),;:]')
//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pep8.py:110: FutureWarning: Possible nested set at position 1
EXTRANEOUS_WHITESPACE_REGEX = re.compile(r'[[({] | []}),;:]')
//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pep8.py:110: FutureWarning: Possible nested set at position 1
EXTRANEOUS_WHITESPACE_REGEX = re.compile(r'[[({] | []}),;:]')
//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pep8.py:110: FutureWarning: Possible nested set at position 1
EXTRANEOUS_WHITESPACE_REGEX = re.compile(r'[[({] | []}),;:]')
4 workers [278 items]
........................................................................ [ 25%]
........................................................................ [ 51%]
........................................................................ [ 77%]
.............................................................. [100%]taxcalc/tests/test_calculator.py::test_calculator_mtr is slower than the current benchmark by 7568.432 ms
taxcalc/tests/test_calculator.py::test_ID_HC_vs_BS is slower than the current benchmark by 1146.178 ms
taxcalc/tests/test_calculator.py::test_ID_StateLocal_HC_vs_CRT is slower than the current benchmark by 1678.292 ms
taxcalc/tests/test_calculator.py::test_ID_RealEstate_HC_vs_CRT is slower than the current benchmark by 1882.211 ms
taxcalc/tests/test_policy.py::TestAdjust::test_apply_cpi_offset is slower than the current benchmark by 1326.902 ms
taxcalc/tests/test_taxcalcio.py::test_output_options is slower than the current benchmark by 3920.533 ms
taxcalc/tests/test_taxcalcio.py::test_no_tables_or_graphs is slower than the current benchmark by 1019.528 ms
taxcalc/tests/test_calcfunctions.py::test_EITCamount[test_tuple0-6660] is slower than the current benchmark by 2047.391 ms
taxcalc/tests/test_calculator.py::test_qbid_limit_switch is slower than the current benchmark by 20668.816 ms
taxcalc/tests/test_calculator.py::test_calc_all_benefits_amounts is slower than the current benchmark by 1168.572 ms
taxcalc/tests/test_consumption.py::test_consumption_response is slower than the current benchmark by 1126.08 ms
taxcalc/tests/test_cpscsv.py::test_agg is slower than the current benchmark by 13421.672 ms
taxcalc/tests/test_policy.py::test_index_offset_reform is slower than the current benchmark by 1045.506 ms
taxcalc/tests/test_reforms.py::test_round_trip_reforms[2019] is slower than the current benchmark by 2254.137 ms
taxcalc/tests/test_reforms.py::test_round_trip_reforms[2020] is slower than the current benchmark by 1798.684 ms
taxcalc/tests/test_reforms.py::test_round_trip_reforms[2021] is slower than the current benchmark by 2165.293 ms
taxcalc/tests/test_taxcalcio.py::test_write_doc_file is slower than the current benchmark by 1906.345 ms
taxcalc/tests/test_calculator.py::test_reform_documentation is slower than the current benchmark by 1387.016 ms
taxcalc/tests/test_calculator.py::test_distribution_tables is slower than the current benchmark by 21282.246 ms
taxcalc/tests/test_calculator.py::test_ce_aftertax_income is slower than the current benchmark by 1155.073 ms
taxcalc/tests/test_policy.py::TestAdjust::test_multiple_cpi_swaps2 is slower than the current benchmark by 1060.249 ms
taxcalc/tests/test_policy.py::TestAdjust::test_adj_CPI_offset_and_index_status is slower than the current benchmark by 1213.617 ms
taxcalc/tests/test_policy.py::TestAdjust::test_cpi_offset_does_not_affect_wage_indexed_params is slower than the current benchmark by 1273.254 ms
taxcalc/tests/test_reforms.py::test_round_trip_reforms[2022] is slower than the current benchmark by 2263.939 ms
taxcalc/tests/test_reforms.py::test_round_trip_reforms[2023] is slower than the current benchmark by 1899.54 ms
taxcalc/tests/test_reforms.py::test_reform_json_and_output is slower than the current benchmark by 8708.887 ms
taxcalc/tests/test_benefits.py::test_benefits is slower than the current benchmark by 1901.282 ms
taxcalc/tests/test_parameters.py::test_json_file_contents[policy_current_law.json] is slower than the current benchmark by 15853.787 ms
taxcalc/tests/test_taxcalcio.py::test_creation_with_aging is slower than the current benchmark by 2442.26 ms
taxcalc/tests/test_taxcalcio.py::test_ctor_init_with_cps_files is slower than the current benchmark by 6147.77 ms
taxcalc/tests/test_taxcalcio.py::test_graphs is slower than the current benchmark by 20176.115 ms
=============================== warnings summary ===============================
Traceback (most recent call last):
File "//Users/mrh/anaconda3/envs/taxcalc-dev/bin/pytest", line 10, in <module>
sys.exit(console_main())
^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/_pytest/config/__init__.py", line 192, in console_main
code = main()
^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/_pytest/config/__init__.py", line 169, in main
ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pluggy/_hooks.py", line 265, in __call__
return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pluggy/_manager.py", line 80, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pluggy/_callers.py", line 60, in _multicall
return outcome.get_result()
^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pluggy/_result.py", line 60, in get_result
raise ex[1].with_traceback(ex[2])
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pluggy/_callers.py", line 39, in _multicall
res = hook_impl.function(*args)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/_pytest/main.py", line 318, in pytest_cmdline_main
return wrap_session(config, _main)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/_pytest/main.py", line 306, in wrap_session
config.hook.pytest_sessionfinish(
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pluggy/_hooks.py", line 265, in __call__
return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pluggy/_manager.py", line 80, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pluggy/_callers.py", line 55, in _multicall
gen.send(outcome)
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/_pytest/terminal.py", line 867, in pytest_sessionfinish
self.config.hook.pytest_terminal_summary(
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pluggy/_hooks.py", line 265, in __call__
return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pluggy/_manager.py", line 80, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pluggy/_callers.py", line 60, in _multicall
return outcome.get_result()
^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pluggy/_result.py", line 60, in get_result
raise ex[1].with_traceback(ex[2])
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pluggy/_callers.py", line 34, in _multicall
next(gen) # first yield
^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/_pytest/terminal.py", line 883, in pytest_terminal_summary
self.summary_warnings()
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/_pytest/terminal.py", line 997, in summary_warnings
maybe_location = collapsed_location_report(message_reports)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/_pytest/terminal.py", line 979, in collapsed_location_report
location = w.get_location(self.config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/_pytest/terminal.py", line 337, in get_location
relpath = bestrelpath(config.invocation_params.dir, absolutepath(filename))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/_pytest/pathlib.py", line 770, in bestrelpath
reldest = dest.relative_to(base)
^^^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/pathlib.py", line 730, in relative_to
raise ValueError("{!r} is not in the subpath of {!r}"
ValueError: '//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/fsspec/registry.py' is not in the subpath of '/Users/mrh' OR one path is relative and the other is absolute.
make: *** [pytest-cps] Error 1
(taxcalc-dev) Tax-Calculator% >....
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pluggy/_result.py", line 60, in get_result
raise ex[1].with_traceback(ex[2])
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/pluggy/_callers.py", line 34, in _multicall
next(gen) # first yield
^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/_pytest/terminal.py", line 883, in pytest_terminal_summary
self.summary_warnings()
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/_pytest/terminal.py", line 997, in summary_warnings
maybe_location = collapsed_location_report(message_reports)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/_pytest/terminal.py", line 979, in collapsed_location_report
location = w.get_location(self.config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/_pytest/terminal.py", line 337, in get_location
relpath = bestrelpath(config.invocation_params.dir, absolutepath(filename))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/_pytest/pathlib.py", line 770, in bestrelpath
reldest = dest.relative_to(base)
^^^^^^^^^^^^^^^^^^^^^^
File "//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/pathlib.py", line 730, in relative_to
raise ValueError("{!r} is not in the subpath of {!r}"
ValueError: '//Users/mrh/anaconda3/envs/taxcalc-dev/lib/python3.11/site-packages/fsspec/registry.py' is not in the subpath of '/Users/mrh' OR one path is relative and the other is absolute.
make: *** [pytest-cps] Error 1
from tax-calculator.
jdebacker
commented on September 22, 2024
@martinholmer asks:
First question:
Why are we timing the tests and where do the benchmark test execution times come from?
The performance benchmarking was introduced in PR #2570 to provide automated profiling of changes in performance for new PRs. At the time, work was being done to consider vectorized code with Dask arrays rather than Numba-jitted functions, so this was especially relevant. The test execution times are reported from pytest and gathered and formatted in Line 138 of conftest.py.
Second question:
Why am I getting this pytest error? Does it have to do with the timing of the tests?
I do not know. It looks like all the tests in pytest passed for you. I don't know what to mark of the warnings summary. I actually have a much longer list of warnings, with notes about future deprecations in the Pandas API, etc.
from tax-calculator.
martinholmer
commented on September 22, 2024
@jdebacker said in issue #2711:
The performance benchmarking was introduced in PR #2570 to provide automated profiling of changes in performance for new PRs. At the time, work was being done to consider vectorized code with Dask arrays rather than Numba-jitted functions, so this was especially relevant.
OK. Is a change from numba to dask still under consideration? If not, can we somehow skip the pytest_sessionfinish because it keeps leaving a file on my disk that is not part of the repository. This is just harassment of developers whose only sin is to have a slower computer than the one used to establish the "benchmark" timings. After running "make pytest-cps", running "git status" produces this:
Untracked files:
(use "git add <file>..." to include in what will be committed)
taxcalc/tests/test_stats_current.csv
What am I supposed to do with this untracked file?
from tax-calculator.
Related Issues (20)
- Payroll tax validation HOT 7
- credits for child tax & other dependents tax calculation of the year 2021 HOT 18
- other dependent credit missing from iitax HOT 6
- Add `openpyxl` to environment file HOT 1
- Create new 3.5.0 release HOT 2
- Are root permissions needed in Linux? HOT 3
- Add `paramtools` to `setup.py` HOT 1
- Codecov token still missing HOT 1
- Still have Codecov token problem
- TCJA extension reform HOT 9
- Phase-out Standard Deduction HOT 1
- Remove TAXSIM-27 validation? HOT 1
- tc cli apparently will not process years after 2034 HOT 1
- Fix CDCC phase-out logic
- Correct CARES charity deduction for nonitemizers HOT 1
- Four incorrect PT_rt? policy parameter values for 2020
- Issues with `write_decile_table()` HOT 2
- The domestic production deduction repeal in TCJA should be permanent
- Add `pyproject.toml` file
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tax-calculator.