Add ability to save Task Diagnostic data about pulpcore HOT 13 OPEN

Saving the data as a pulp_file artifact I think is a great idea. That would be even easier for us to pull the data back out then. So would that be:

1. pulp admin creates a pulp-file repo

2. pulp admin configures the `TASK_DIAGNOSTIC_FILE_REPO_URL` equal to its URL? or maybe `TASK_DIAGNOSTIC_FILE_REPO_ID` equal to its UUID?

What do you think?

Adding a file to that repository requires a task. That may create an infinite loop.

from pulpcore.

Time for dogfooding: Let's put any type of report in artifacts.

from pulpcore.

Maybe use pulp_file to save this data.
It's gonna use the Storage that Pulp is using.

from pulpcore.

Which things we're getting from the diagnostics? Can we get the same information using other ways, like container metrics, or OTEL?

from pulpcore.

Saving the data as a pulp_file artifact I think is a great idea. That would be even easier for us to pull the data back out then. So would that be:

1. pulp admin creates a pulp-file repo
2. pulp admin configures the TASK_DIAGNOSTIC_FILE_REPO_URL equal to its URL? or maybe TASK_DIAGNOSTIC_FILE_REPO_ID equal to its UUID?

What do you think?

from pulpcore.

I see the value of storing the output files solely in artifacts because those could be saved without depending on the tasking system. Also they will automatically be picked up by the orphan clean up.

from pulpcore.

Saving them as an artifact sounds even better for all the reasons mentioned. Also it would avoid the admin even having to create a dedicated pulp_file repo.

When a task runs, how can I know how to fetch that saved artifact? If that's easy then I'm +1 on this.

from pulpcore.

I thought about this some more and what would be nice is if the Task got a created_resources entry in it to the Artifact URL. Is that an option?

from pulpcore.

We should be able to at least to log the href of the created artifact whenever task_diagnostics is on.

from pulpcore.

Logging would work. Having it on the task is more beneficial for our use for a few reasons:

• Sometimes the logs can be hard to get a hold of on the hosted installations because it's not a system you just log into
• We'll leave this enabled on staging, and we want our hosted users to be able to get their cProfile data in a self-service way (without having to contact us for each cProfile they want).

from pulpcore.

@bmbouter I agree that having it directly on the task is more transparent, just we never placed artifacts nor content as created_resource, only those resources that require locking, like repos, remotes, distributions, etc

from pulpcore.

@ipanova I agree and I'm also a little uneasy setting that precedent here. Is there another option to have that info added to the task somehow?

from pulpcore.

We discussed at pulpcore meeting today. Here's my summary of what I think the plan is.

Let's implement this setting to save the bare Artifact (not a file content unit, not in a file repo) and just have it log for now. We can look at improving the usability later as a separate issue. This avoids us having to set the precident adding an Artifact to the created_resources data of a task.

@pulp/core is this right?

from pulpcore.