Comments (2)
This is actually a common practice in session-based recommendation. The same setting was adopted in some previous works such as STAMP, SR-GNN, and FGNN, and in some recently published works such as DHCN and TAGNN. We could even find the same setting in other fields of recommender systems, such as LightGCN and NGCF.
Despite the popularity of our setting, I agree with you that in principle, we should choose the best model on the validation set and evaluate the model on the test set. The common approach is to first extract a most recent fraction of the data as the test set, and then extract a most recent fraction of the remaining data as the validation set. But this approach may also have a problem, i.e., the distributions of the validation set and the test set are different because their collection times are different, so the performance on the validation set does not reveal the true performance on the test set. I guess this is part of the reason some existing methods did not adopt the correct-in-principle approach. I think a better approach is to first extract a most recent fraction of the data and then randomly split it into the validation and test sets. But I did not think of this approach when we published this work.
In conclusion, I think our setting is OK because it is very common. Besides, it is used for all models in our experiments. The comparison of models' performance is fair.
You could update our code to use the validation set to select the best model. It should be quite easy. However, we would not update our code to make it consistent with our paper.
from lessr.
Still confused by this common practice, but thanks for your reply.
from lessr.
Related Issues (10)
- Question about EOPA HOT 1
- question data preprocessing HOT 8
- question data preprocessing HOT 1
- Gowalla and Last.fm HOT 1
- Baseline code HOT 2
- Issue the request:RuntimeError
- question data preprocessing HOT 1
- Data preprocessing for youchoose HOT 1
- Full form of 'rst' and 'fc'? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lessr.