toolchainlabs / issues Goto Github PK

View Code? Open in Web Editor NEW

0.0 3.0 0.0 1 KB

Public repo just for filing bugs and feature requests

issues's Introduction

This is where you can file bugs and feature requests for Toolchain's services.

issues's People

Contributors

Watchers

issues's Issues

High total runtime for `test`, even with very high cache rate

For larger number of tests, the startup time and overhead of pants in some private repositories can be very high. At the same time, memory usage is unexpectedly high (as first reported in pantsbuild/pants#12662).

Most likely the memory usage and total runtime are related, so pursuing the metrics gathering exercise from pantsbuild/pants#12662 would likely be very valuable in also tracking down the performance issue.

Feature: "raw" workunits JSON

Maybe I want to do more interesting things with the JSON, like various transformations to see flamegraphs or whatever.

Feature: Ability to filter a in BuildSense by targets

We are currently splitting the execution into multiple workflows because we can't fit ./pants test :: into a single workflow.
We want to be able to filter by the target, for example, if I have the following command:

./pants test a/test_a.py b/test_b.py

I want to be able to filter out by test_b.py

JSONDecodeError whenever communicating with Buildsense after acquire-auth

(originally reported as pantsbuild/pants#14222)

Describe the bug

Hi! I've acquired auth but Pants seem to be having a hard time talking with Buildsense.

Any time I run any Pants command it spews out a JSON error:

./pants filter --target-type=python_sources,python_tests :: | xargs ./pants lint
16:06:44.23 [INFO] Completed: Lint with isort - isort succeeded.
Skipped 47 files


16:06:44.87 [INFO] View on BuildSense: https://app.toolchain.com/organizations/grapl-security/repos/grapl/builds/pants_run_2022_01_20_16_06_43_209_0e550a27bfda4f199e846d8ccfd06ecc/
Exception in thread buildsense-reporter:
Traceback (most recent call last):
  File "/home/wimax/.pyenv/versions/3.7.10/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/home/wimax/.pyenv/versions/3.7.10/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/wimax/.cache/pants/named_caches/pex_root/venvs/s/6ae700a8/venv/lib/python3.7/site-packages/toolchain/pants/buildsense/reporter.py", line 338, in _report_loop
    operation = self._build_state.send_report()
  File "/home/wimax/.cache/pants/named_caches/pex_root/venvs/s/6ae700a8/venv/lib/python3.7/site-packages/toolchain/pants/buildsense/state.py", line 151, in send_report
    batch_sent = self._maybe_send_batched_build()
  File "/home/wimax/.cache/pants/named_caches/pex_root/venvs/s/6ae700a8/venv/lib/python3.7/site-packages/toolchain/pants/buildsense/state.py", line 248, in _maybe_send_batched_build
    batched_build = self._local_store.get_upload_batch()
  File "/home/wimax/.cache/pants/named_caches/pex_root/venvs/s/6ae700a8/venv/lib/python3.7/site-packages/toolchain/pants/buildsense/local_store.py", line 86, in get_upload_batch
    return self._create_batch()
  File "/home/wimax/.cache/pants/named_caches/pex_root/venvs/s/6ae700a8/venv/lib/python3.7/site-packages/toolchain/pants/buildsense/local_store.py", line 97, in _create_batch
    batch, files_in_batch = self._get_batched_data()
  File "/home/wimax/.cache/pants/named_caches/pex_root/venvs/s/6ae700a8/venv/lib/python3.7/site-packages/toolchain/pants/buildsense/local_store.py", line 118, in _get_batched_data
    batch_data[key] = {build_file.stem: json.loads(build_file.read_bytes()) for build_file in files_batch}  # type: ignore[assignment]
  File "/home/wimax/.cache/pants/named_caches/pex_root/venvs/s/6ae700a8/venv/lib/python3.7/site-packages/toolchain/pants/buildsense/local_store.py", line 118, in <dictcomp>
    batch_data[key] = {build_file.stem: json.loads(build_file.read_bytes()) for build_file in files_batch}  # type: ignore[assignment]
  File "/home/wimax/.pyenv/versions/3.7.10/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/home/wimax/.pyenv/versions/3.7.10/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/wimax/.pyenv/versions/3.7.10/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

16:06:47.78 [INFO] Completed: Lint with Flake8 - Flake8 succeeded.
16:06:47.80 [INFO] Completed: Lint with Black - Black succeeded.
All done! ✨ 🍰 ✨
185 files would be left unchanged.



✓ Black succeeded.
✓ Flake8 succeeded.
✓ isort succeeded.

The token is fresh and has these permissions:

Pants version
Which version of Pants are you using?
2.8.0

OS
Are you encountering the bug on MacOS, Linux, or both?
linux

Additional info

Command UI trims the command

This includes when I hit the copy button

APIs to enable for downloading trace data

We would like to be able to download trace data from all (or some subset) or our pants runs.

I would like to build a small tool that runs this algorithm, likely on cron-like schedule every few minutes:

builds = list_recent_builds()
for build in build:
  trace_data = get_trace_data(build)
  upload_trace(trace_data)

I would be happy to have to keep track of which builds we have seen, but efficient pagination (show me builds since id x or since time t) would be useful.

I can see all the data we want in the web app already and even tried to do this in a hacky way–

$ export TOOLCHAIN=`cat .pants.d/toolchain_auth/auth_token.json  | jq  -r .access_token`
$ curl -i -H "Authorization: bearer $TOOLCHAIN"  https://app.toolchain.com/api/v1/repos/color/color/builds/
HTTP/2 403 
...

N/A plan, no usage data and "customer record doesn't exist" for new trial sign-up

I've signed my org stellargraph up to a Toolchain trial (finally, months after I first approached you! 😅 ), and noticed that our org page https://app.toolchain.com/organizations/stellargraph/ has missing data on it, presumably where I'm meant to see details about the trial plan and usage:

Clicking the 'Manage plan' button gives that "customer record doesn't exist" error pop-up.

Buildsense for the one activated repo still seems to work though.

BuildSense reports skipped tests as failing tests

What happens:
We have python tests that are skipped, BuildSense reports them as failing

What should happen:
The tests should be reported as skipped

BuildSense time reporting for runtimes is incorrect

I have examined a few traces in BuildSense and found that BuildSense reported time and the time reported on the Zipkin traces have big disparities, I have seen BuildSense reporting ~20 seconds and the Zipkin trace showing ~90 seconds

Can you link here on the UI?

(Title says it all!)

Feature: BuildSense add trace visualization

Today, in order to visualize a trace from pants I need to download the trace, trim it with a custom jq command and load it into zipkin. It makes debugging the build very cumbersome.

I would like to be able to visualize the trace within BuildSense

More System info in "Local Machine Info"

C'mon, blast me with info!

Memory
Processor
Disk capacity?
OS type/name/version
GPU? (We have CUDA tests conditionally running so why not)

Just drown me in info ;)

Feature: BuildSense should link back to the relevant CI

What happens:
If I open a build in BuildSense I can't get from it back to the relevant CI pipeline, there's no context propagated that can help me find it. The branch / actor helps, but it's still requires me to search many workflows until I find the correct one.

What I would like to happen:
I would like to have a link in BuildSense back to the CI pipeline that executed it

Feature: BuildSense test trends

Without care over time test suites get slower, I would like to be able to view the test suite performance as a whole and per test target performance.

I would also like to write alerts and notify relevant team members about the relative performance of their tests

toolchainlabs / issues Goto Github PK

issues's Introduction

issues's People

Contributors

Watchers

issues's Issues

Recommend Projects

Recommend Topics

Recommend Org