[Proposal] Index Environment Metadata for every benchmark about benchmark-wrapper HOT 4 CLOSED

Talked about this with @whitleykeith on slack. Since backpack already collects all the data prior to any run we could simply have it write its json output to a shared storage or just the bits we care about to redis and then have snafu read it in. It would be a pretty trivial change to backpack and snafu then. thoughts?

from benchmark-wrapper.

My only concern with backpack is that it seems like a pretty heavy dependency. Would the automotive team be able to use backpack for environment information?

Also, what sort of information are we looking to collect?

I like the idea of having snafu/environments for a location of Python modules that pull env metadata. Another idea that might work, since some env collection could be done in straight-up bash, is the following:

• Create snafu/environments as a collection of tiny executable scripts which take no arguments and output environment data in the form of KEY=VALUE, with each key-value pair on a separate line. If no key-value pair was found in the output, or if the return-code is non-zero, then assume the script isn't applicable for the environment.
• Create a module called environment.py which runs each of these bash scripts one by one, collects the output, and saves the key-value pairs to a dataclass which we can then export.

From this perspective, we can do metacollection using bash, perl, python, whatever. It'll be super lightweight, easy to maintain, easy to test and have minimal external dependencies.

But at the disadvantage that we now have to maintain code that does the same thing as backpack.

from benchmark-wrapper.

My only concern with backpack is that it seems like a pretty heavy dependency. Would the automotive team be able to use backpack for environment information?

They would never use backpack -- backpack is specific to OCP. They would use the underlying Ansible roles which is Stockpile.

Also, what sort of information are we looking to collect?

I like the idea of having snafu/environments for a location of Python modules that pull env metadata. Another idea that might work, since some env collection could be done in straight-up bash, is the following:

• Create snafu/environments as a collection of tiny executable scripts which take no arguments and output environment data in the form of KEY=VALUE, with each key-value pair on a separate line. If no key-value pair was found in the output, or if the return-code is non-zero, then assume the script isn't applicable for the environment.
• Create a module called environment.py which runs each of these bash scripts one by one, collects the output, and saves the key-value pairs to a dataclass which we can then export.

From this perspective, we can do metacollection using bash, perl, python, whatever. It'll be super lightweight, easy to maintain, easy to test and have minimal external dependencies.

But at the disadvantage that we now have to maintain code that does the same thing as backpack.

IMHO This is a step backwards. We already do this today. Check out Stockpile and Scribe.

from benchmark-wrapper.

My concern with Scribe was that it seemed like a large dependency, which was where that bash idea came from, but if we are willing to do use it then all the merrier. My bad on the confusion.

from benchmark-wrapper.