ui and/or command line to merge multiple json results about fortio HOT 12 OPEN

listing down a few assumptions here for the cli mode

• sample usage:

$ fortio merge file1.json file2.json ...

this should create a merged file (what should it be named?) with all the input result files. (file paths are relative and output file will be created in the same directory)

approach:

• parse os.Args for .json files
• iteratively merge each file
• how to merge:
  • Labels (append all? choose the first file's label as the only label)
  • merge StartTime (earliest wins)
  • merge RequestedQPS (average? how to handle differences?)
  • merge RequestedDuration (maximum)
  • ActualQPS (average)
  • ActualDuration (average)
  • NumThreads (sum)
  • Version (first file's version)
  • RetCodes (dependent on identifying the type of result)
  • HistogramData
    • convert to Histogram and call Merge? is the conversion possible?
    • create MergeHistogramData and aggregate values in common buckets

open questions:

1. how to determine if results are mergeable? (eg assuming similar start times)
2. should the type of results be specified in the saved json file? (http/grpc)

from fortio.

Thanks for looking into this!

this should create a merged file (what should it be named?) with all the input result files. (file paths are relative and output file will be created in the same directory)

if -a (current autosave) is specified it can create a new ID which then turns into a file name
otherwise the merged output could be stdout (like -json - does today)

or you have to specify -json name

probably it should warn if values are very different (like merging 1 requested qps with 100)
we could use "merged" as placeholder for string types or even null out some fields in merged json
(would have to check they are still graphable, potentially fixing bugs in graphing conversion)

ActualDuration (average)

sum would be accurate here, not avg

NumThreads (sum)

No, avg, or replaced by a list of numbers when they are not ==

ActualQPS (average)

not really, should be recalculated - ie weighted avg - 1s run at 10kqps + 10s run at 5qps is not an 11s run at 7.5kqps

RetCodes (dependent on identifying the type of result)

nope, just merged, it's a map

should the type of results be specified in the saved json file? (http/grpc)

yes I think I mentioned that in #176 (comment)

from fortio.

my initial guess at this functionality was to allow merge of result files produced from a distributed client mode. in that case, labels, version.. metadata wouldn't change much. and one can also assume similar offset/multiplier for producing the histogram data.

In that scenario, some of the suggestions aren't adding up eg.

NumThreads (sum)
No, avg, or replaced by a list of numbers when they are not ==

shouldn't this just be the cumulative number of threads used in the distributed client mode?

similarly ActualDuration can be max of all the durations reported by the clients.

from fortio.

You're right I'm actually wrong about how to merge the duration... it depends

I was thinking of merge of back to back tests

it's like intervals of start time + duration and we can use earliest starttime -> last end time instead
but there could be gaps

from fortio.

i am running into a few technical difficulties that i can use some help with.
priority: low

given a fortio result file, how to read and unmarshal it as http/grpc results?

option 1:

• unmarshal it as periodic.RunnerResults
  • lose mode-specific info

option 2:

• based on a flag/convention unmarshal it specifically to a [GRPC/HTTP]RunnerResult
  • lots of code duplication

option 3:

• multiphase unmarshal?
• unmarshal common attributes to periodic.RunnerResults
• based on mode info present in common attributes, parse into HTTP or GRPC results
  • i am not sure if this is a recommended pattern

also, one way to avoid this will be to extract RetCodes and bring it into the common RunnerResults. any suggestions on this?

from fortio.

definitively 3 I guess

we should put enough information that it is (easily/cleanly) reversible

it's an O/R mapping issue, I have to think (or google around) how "subclass" mapping to json works in the reverse (deserialization) case but it's solvable(/solved)

from fortio.

thanks for showing the light at the end of the tunnel :) i will try to dig more for option 3

from fortio.

it's train :-) (the light)

from fortio.

Has anyone lately looked into this? I'm currently looking at distributed client mode and would be needing to combine multiple result files. Thanks in advance.

from fortio.

it’s all yours to tackle and contribute! thx!

from fortio.

Is there any way to convert a HistogramData to Histogram

Currently, the only way I can think of is to include Offset and Divider (both variable in Histogram) in HistogramData to recover the Histogram from HistogramData.

from fortio.

offset is rarely ever set so you could assume it's 0 and you could also guess/calculate both looking at the bucket widths

we can add the info if it would help but what would it help with ?

in general if you do a distributed client run, I'd expect you'd set the same offset and divider on all client (why wouldn't you?) or that your merging code would be generic enough to handle non overlapping buckets one way or another?

from fortio.