Comments (5)
The issue is that dumping the benchmark into a JSON file for the many benchmarks I want to run, which are almost entirely identical except for a different combinations of arguments, would mean I would get tons of JSON files saved.
On top of that, this looping issue would actually write the JSON file for each process that pyperf.Runner
is given, so 20 processes would mean that the output JSON file would be written 21 times, once for each process run and one last time where all the runs are aggregated.
from pyperf.
Maybe write:
if __name__ == "__main__":
runner = pyperf.Runner()
runner.bench_func('sleep', func)
from pyperf.
I have tried that, but the output remains the same. Just to make sure, this is what that code looks like:
#!/usr/bin/env python3
import pyperf
import time
def func():
time.sleep(0.001)
if __name__ == "__main__":
runner = pyperf.Runner()
results = runner.bench_func('sleep', func)
print(results)
My workaround for this weird bug is to know how many processes are being used (either set by the user or use the default of 20) and check if len(result.get_runs()) == procs
which looks like so:
#!/usr/bin/env python3
import pyperf
import time
def func():
time.sleep(0.001)
if __name__ == "__main__":
procs = 20
runner = pyperf.Runner(processes=procs)
results = runner.bench_func('sleep', func)
if len(result.get_runs()) == procs:
print(f"Mean: {results.mean()}")
print(f"Median: {results.median()}")
But from my understanding of the overall code, pyperf
calls new instances of the python to run the same program file, which is why it ends up printing out multiple times rather than after the benchmark is completed.
from pyperf.
An update regarding this is that this issue also makes it mildly difficult to output all the results into a single output file. Here's an example:
#!/usr/bin/env python3
import pyperf
import time
import csv
from statistics import mean, median
def func():
time.sleep(0.001)
if __name__ == "__main__":
procs = 20
runner = pyperf.Runner(processes=procs)
my_results = {}
for i in range(1,11):
result = runner.bench_func('sleep', func)
if len(result.get_runs()) == procs:
my_results[i] = list(result.get_values())
with open("output.csv", "w", "newline="") as my_file:
headers = ["Loop", "Mean", "Median"]
writer = csv.DictWriter(my_file, fieldnames=headers)
writer.writeheader()
for k in my_results.keys():
writer.writerow(
{
"Loop": k,
"Mean": mean(my_results[k]),
"Median": median(my_results[k]),
}
)
Even though I have the if len(result.get_runs()) == procs
block above to force the program to wait until all the results are available, this doesn't prevent the program from continuing onwards to later code. I also have to change the if len(result.get_runs()) == procs
block to check if result is None
since it will be None until any benchmark result is returned, which is not always the case in the loop. This is the fixed version of the code:
#!/usr/bin/env python3
import pyperf
import time
import csv
from pathlib import Path
from statistics import mean, median
def func():
time.sleep(0.001)
if __name__ == "__main__":
procs = 20
runner = pyperf.Runner(processes=procs)
my_results = {}
for i in range(1,11):
result = runner.bench_func('sleep', func)
if result is None:
pass
elif len(result.get_runs()) == procs:
my_results[i] = list(result.get_values())
open_mode = "w"
if Path("output.csv").exists():
open_mode = "a"
csv_file = open("output.csv", open_mode, newline="")
headers = ["Loop", "Mean", "Median"]
writer = csv.DictWriter(my_file, fieldnames=headers)
if open_mode == "a":
writer.writeheader()
for k in my_results.keys():
writer.writerow(
{
"Loop": k,
"Mean": mean(my_results[k]),
"Median": median(my_results[k]),
}
)
This is kind of cumbersome. I'm still trying to understand the underlying code for pyperf to see why this is occurring, but I'd want to look into creating a wait
command (similar to asyncio
's async/await) where all runs have to complete before moving on with the rest of the program.
from pyperf.
That's a surprising way to use pyperf. Why not writing results in a JSON file and then load the JSON to process it?
https://pyperf.readthedocs.io/en/latest/api.html#BenchmarkSuite.load
from pyperf.
Related Issues (20)
- Support collecting cProfile data from a benchmark HOT 1
- Increase default running time per process? HOT 7
- Having trouble while using pypy: pyperf is much slower than timeit HOT 4
- Allow for warmup in PEP659 HOT 7
- Pyperf API examples fail when running from a REPL HOT 2
- Propagating PYTHONPATH in tests HOT 2
- Support custom loop factory in bench_async_func
- Instability and slowness after using `pyperf system tune` HOT 2
- Can process run in parallel? HOT 4
- Display additional information through table.
- Programmatically define output file for bench_func HOT 1
- bench_func returning None for all but the first benchmark
- How to programmatically get the output for timeit() or bench_func()? HOT 9
- Add Heirarchical Performance Testing (HPT) technique to `compare_to`? HOT 3
- 2.6.2: sphinx warnings `reference target not found`
- Disable pyperf for free-threaded build for a while HOT 1
- Include `CONFIG_ARGS` in Metadata HOT 3
- Benchmark with disabled GIL build and -Xgil=1 isn't enabling GIL HOT 6
- `warmups` not a supported parameter for Runner anymore
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pyperf.