Training GP on outputs from the last hidden layer about keras-gp HOT 3 CLOSED

Hi. Yes, I like the way of thinking of generalization of the last layer in the space of high-level features obtained by transforming the inputs. And this is commonly used across deep learning applications -- you pretrain your network on a large general purpose dataset and then use a part of it to construct a new model for another dataset you care about. As long as the data shares some semantics, pretraining could be a useful initialization (it's hard to quantify this theoretically though, people just do it through trial and error).

Note that usually you still fine-tune the entire model after pretraining (with a much lower learning rate to eliminate "forgetting" of the initial state), which typically gives better performance than just training the final layer (we have tried both in this paper, and see more results in the deep kernel learning paper). Trying GP layers for transfer learning across datasets is something we haven't tried/thought about and is potentially worth looking into.

from keras-gp.

Thank you for your response!

You mention the transfer learning case - when I use a different dataset. As in your paper I want to use the same train data for a complete model training (\w linear layer) and for a separate GP training routine. With my training-data X I would calculate the output of the last hidden layer H and with (H, y) train the GP model.

During testing I could use the output of the linear layer, prediction of the GP and the confidence of the GP. The motivation is to use the GP outputs in the backend to verify the model confidence from time to time.

from keras-gp.

That sounds like an interesting idea to try! I am closing the issue. For general questions of this sort, please consider sending an email instead opening an issue.

from keras-gp.