Scripts packaging and distribution network about bcc HOT 4 OPEN

Can I close this ticket? I still think it's over engineering for most people...

from bcc.

With all respect, I don't think this is a good idea for the following reasons:

• It's creating an extra interface that people need to learn, that overlaps with more intuitive methods. Info on a script? Man page. What scripts use task-switch? grep.
• Running scripts via "bpf-run" is not conventional. Convention is to have separate stat tools which do one thing and do it well. iostat, mpstat, vmstat, .... This is a model that has worked quite well when doing tracing scripts as well.
• Running scripts via "bpf-run" may restrict argument usage. Does bpf-run have other options? How can my tracing scripts then have full control when wrapped by bpf-run?
• Having infrastructure where people can submit bundles of scripts for others to use: who is going to approve of the scripts? Tracing tools often make it easy to write scripts that appear to work and appear to be useful, but can be very difficult to properly test and turn these into the script that should be supported. It's very easy to gather a collection of garbage (other projects have taken this route), which would hurt the bcc project. For more background, see mistake 19 in http://bdgregg.blogspot.com/2013/09/dtracetoolkit-0xx-mistakes.html.

Having a website that resembles a search engine to find scripts and search inside scripts may be useful, provided it made it clear all scripts were unsupported.

from bcc.

Thanks for the feedback. This idea I had was extension of the demonstration we had of bpf-run in BPF event yesterday at LinuxCon actually coupled with some discussion about how to create and distribute scripts.

It's creating an extra interface that people need to learn, that overlaps with more intuitive methods. Info on a script? Man page. What scripts use task-switch? grep.

Running scripts via "bpf-run" may restrict argument usage. Does bpf-run have other options? How can my tracing scripts then have full control when wrapped by bpf-run?

My suggestion is to not have bpf-run as the only means of using BPF but just as an extension for those folks who would just like to use the pre-built and 'validated' scripts without altering them. I think bpf-run or just plain bpf or something similar as a command won't hurt as it can be the way for new users to just use the default options. For power users who can write and alter their scripts, we can just allow them to bpf-get the source of scripts easily and alter and repackage them for their purpose.

Running scripts via "bpf-run" is not conventional. Convention is to have separate stat tools which do one thing and do it well. iostat, mpstat, vmstat, .... This is a model that has worked quite well when doing tracing scripts as well.

Yes, I agree with this. I think you are in a better position to know what is more relevant.

Having infrastructure where people can submit bundles of scripts for others to use: who is going to approve of the scripts? Tracing tools often make it easy to write scripts that appear to work and appear to be useful, but can be very difficult to properly test and turn these into the script that should be supported.

Just like in packaging for distributions we can have a review process etc. to overcome this. Soon, over some time, we would have a large collection of scripts just like yours which would be a bit easy to manage and use. In one of your presentations you mention that the script users are numerous but the creators are a few. This can be a good opportunity to have a script distribution system for those creators so that they can reach the numerous script users easily. When I want to try SystemTap, it would really help me as a user to just look at those examples, right from easy to elaborate, well classified and organized from which I could pick some, alter and try them or just directly run them - right from where I am working. The problems which I see is the considerable effort which would be put in validating the scripts and making sure they work always. This I agree can be counter-productive for some.

Even if this is a bad idea I would just stress on the fact that we could do these two things - teach people how to use BCC and BPF scripts from ground up, and/or distribute scripts and just lightly hint users how to modify them and carry onwards. We could also do both. Python has a successful way of distributing packages with the optional pip which I find useful. But this does not stop developers from creating their own python packages or taking source of the packages and altering it.

Tracing tools often make it easy to write scripts that appear to work and appear to be useful, but can be very difficult to properly test and turn these into the script that should be supported. It's very easy to gather a collection of garbage (other projects have taken this route), which would hurt the bcc project. For more background, see mistake 19 in http://bdgregg.blogspot.com/2013/09/dtracetoolkit-0xx-mistakes.html.

This is a very nice post :) Thanks. I would say that we can have only validated scripts. As they have followed the review process, they would not ruin the experience. The unsupported ones of course would be 'unrated' on the distribution network (like an extra respository installed manually etc) As I said problem is to manage these contributions and it would not be a one man task.

Apart from this, I would like to say to BCC devs that Brendan's experience is more relevant here as I am just a beginner and voicing out my thoughts and I may be really wrong.

from bcc.

I've been thinking about this request quite a bit, and will continue to think about it some more. I will get around to documenting the formal proposal here, but in a nutshell I would tend to take more of the purist packaging approach when possible. That is, keep everything modular, with simple to follow dependencies, and keep as close to the distribution of choice's packaging structure as possible. To achieve that, the repo and directory structuring of bcc first needs formalization and cleanup, which is why for the time being I will spend some time on that legwork. Thus, you will see some things like #166 .

from bcc.