Stuff that needs doing about abstractgps.jl HOT 8 CLOSED

One large area that we've really not covered so far is benchmarking. It would be very helpful to have a systematic suite of both "unit benchmarks" and "integration benchmarks" that test the details of particular bits of the package and complete pipelines. Currently I have no idea how far behind (or ahead) of GPflow / GPyTorch we are, even for vanilla GP regression.

from abstractgps.jl.

I am not sure if benchmarks should be a part of this package or not. IMO at least workflow benchmarks should either be a separate package or be part of the the higher level abstraction package (whatever we decide to name it).

I am unsure what would constitute as "unit benchmarks". Are you referring to functions such as posterior and approx_posterior? If so, is it worthwhile to benchmark them separately in this package or just have them as part of workflow benchmarks ?

If were to draw comparisons with GPflow / GPyTorch, we would have to first specify the parallels between them and our package as our API is quite different.

from abstractgps.jl.

Is your goal to use this package as a framework for writing packages like Stheno and AugmentedGPs? And if so, are you planning to refactor those packages accordingly?

EDIT:
I don't want to sound offtopic, I'm asking in order to understand the overall scope of AbstractGPs.

from abstractgps.jl.

I agree with @willtebbutt , the package starts to be pretty much complete, we should actually release a 1.0!

The one package that requires more love is KernelFunctions.jl, where there is a large collection of issues to solve :D

Concerning the use for other packages, I am in the process of refactoring a bit AugmentedGP.jl to make it more AbstractGPs.jl compatible

from abstractgps.jl.

Ah maybe something missing would be more testing with the different ADs like in KernelFunctions. I think the MeanFunctions are still uncompatible with Zygote

from abstractgps.jl.

Is your goal to use this package as a framework for writing packages like Stheno and AugmentedGPs? And if so, are you planning to refactor those packages accordingly?

This is precisely on-top :) Stheno and TemporalGPs will both be moved over to AbstractGPs / KernelFunctions once it's sufficiently stable.

Ah maybe something missing would be more testing with the different ADs like in KernelFunctions. I think the MeanFunctions are still uncompatible with Zygote

Very good point.

I am not sure if benchmarks should be a part of this package or not. IMO at least workflow benchmarks should either be a separate package or be part of the the higher level abstraction package (whatever we decide to name it).

I was imagining having a directory called benchmarks (at the same level as src, test, docs etc) that's got a Project.toml and Manifest.toml that systematically breaks down and benchmarks this package. Do you think that this might be problematic?

If were to draw comparisons with GPflow / GPyTorch, we would have to first specify the parallels between them and our package as our API is quite different.

Good point. This is where I was imagining that "integration" benchmarks would come in -- I'm imagining these being things like "learn the parameters of a zero-mean GP with a exponentiated-quadratic kernel to fit dataset X using exact inference", as opposed to "unit" benchmarks like "evaluate the log marginal likelihood of a GP / its gradient as a function of dataset size" or something / "kernelmatrix evaluation for some kernel".

The kind of thing that I have in mind is something like what I started doing in Stheno.jl but never got around to finishing using this package.

Obviously I'm open in terms of what we actually do, but it seems to me that we really ought to start tracking performance so that we can know when things regress.

I agree with @willtebbutt , the package starts to be pretty much complete, we should actually release a 1.0!

Also a good point. I'm tempted to say that we should hold off until AugmentedGPs, Stheno, and TemporalGPs use the abstractions here -- I would prefer to hold off commiting to the abstractions until we know that they're not really problematic for some reason. But I agree that we should be able to tag 1.0 within the next few months.

from abstractgps.jl.

For benchmarks, I would also approve having them here. A while ago I played with PkgBenchmarks.jl which I quite liked. But I did not find any nice workflow to automate the tasks...
The problem is that one can rely on Travis or Github actions to estimate correctly any improvement or regression

from abstractgps.jl.

This is definitely out of date

from abstractgps.jl.