Create time-splits for Materials Project for percent matched notion of best about xtal2png HOT 6 CLOSED

Create two time-splits, one based on MPIDs and one based on ICSD earliest pub year (based on two suggestions linked above).

from xtal2png.

Useful: sklearn.model_selection.TimeSeriesSplit

from xtal2png.

By default TimeSeriesSplit train/test indices for a total of 10 compounds would look like:

TRAIN: [0 1 2 3 4] TEST: [5]
TRAIN: [0 1 2 3 4 5] TEST: [6]
TRAIN: [0 1 2 3 4 5 6] TEST: [7]
TRAIN: [0 1 2 3 4 5 6 7] TEST: [8]
TRAIN: [0 1 2 3 4 5 6 7 8] TEST: [9]

Considering ignoring test_index and using np.setdiff to make the test indices include "everything else" as in everything other than the train_index-s. Or could leave it as-is, but seems kind of strange not to use all the data. Makes sense why TimeSeriesSplit is set up this way, though.

from xtal2png.

Alternative:

TRAIN: [0 1 2 3 4] TEST: [5 6 7 8 9]
TRAIN: [0 1 2 3 4 5] TEST: [6 7 8 9]
TRAIN: [0 1 2 3 4 5 6] TEST: [7 8 9]
TRAIN: [0 1 2 3 4 5 6 7] TEST: [8 9]
TRAIN: [0 1 2 3 4 5 6 7 8] TEST: [9]

Was expected the first to be a 20/80 split, though. Maybe just need to wrap around KFold

from xtal2png.

Here's something very close to what I was originally imagining. It's based on KFold(n_splits=6) + some post-processing:

TRAIN: [0 1] TEST: [2 3 4 5 6 7 8 9]
TRAIN: [0 1 2 3] TEST: [4 5 6 7 8 9]
TRAIN: [0 1 2 3 4 5] TEST: [6 7 8 9]
TRAIN: [0 1 2 3 4 5 6 7] TEST: [8 9]
TRAIN: [0 1 2 3 4 5 6 7 8] TEST: [9]

Default TimeSeriesSplit might not be so bad in that taking the mean and stdDev of the performance metric is more straightforward: no weighting involved. Might be better to go with the more interpretable option, which is the default behavior of TimeSeriesSplit.

@hasan-sayeed @sp8rks feel free to chime in on this one if you have any thoughts.

from xtal2png.

@hasan-sayeed you should be able to use mp_time_split per the usage instructions or one of the examples.

If you run into any issues, lmk at https://github.com/sparks-baird/mp-time-split/issues

from xtal2png.