This line seems to slow the entire training

https://github.com/BirkhoffG/cfnet/blob/6d5c3ec28351de79e575bbded5e67ab1bab48b03/cfnet/module.py#L249-L250

Plugin to #50

There are some outdated links (i.e., some links are still written in cfnet). Please try to fix ALL of them to the correct links.

For example,
https://github.com/BirkhoffG/ReLax/blob/v0.1/relax/data/module.py#L415-L416

Something like:

If monitor_metrics is None:
    # no checkpoint manage

Supporting hyper-parameter searching enables us to properly benchmark the algorithms. This issue is a thread discussing how to support hyperparameter searching in CF explanation methods.

In essence, this problem is a multi-objective problem (i.e., minimizing invalidity and cost).

Some open-sourced libraries of hyper-parameter searching:

• Optuna [doc]
• Ax [doc]
• Dragonfly [link]
• Spearmint [link]
• BoTorch [link]

Currently, we assume pred_fn is a function of only one input x. E.g., it is something like:

pred_fn = lambda x: 2 * x + 1

However, it is possible that user-defined pred_fn takes other arguments.

Hence, I propose

def generate_cf_explanations(
    cf_module: BaseCFModule,
    datamodule: TabularDataModule,
    pred_fn: Callable[[jnp.DeviceArray], jnp.DeviceArray] = None,
    *,
    t_configs=None,
    pred_fn_args: dict=None
)

where inside, we call pred_fn as

pred_fn(x, **pred_fn_args)

This offers additional flexibility for models that are not implemented using our framework.

Seems to run some unnecessary tests (e.g., train some models) during the testing

Pytorch is only needed for loading data. Our library mainly handles tabular data, so data loading would not be a bottleneck to most scenarios. Pytorch Dataloader is overkill for our project in most use cases.

Purpose

Write a drop-in NumpyLoader.

ToDo

Delete the Pytorch Dependency

https://github.com/BirkhoffG/cfnet/blob/24783713dc787cd5b13e70aa483e455c4856198f/settings.ini#L18

https://github.com/BirkhoffG/cfnet/blob/24783713dc787cd5b13e70aa483e455c4856198f/cfnet/datasets.py#L9

Next, modify the following code to make them not inherent Pytorch Dataset and DataLoader:

https://github.com/BirkhoffG/cfnet/blob/24783713dc787cd5b13e70aa483e455c4856198f/cfnet/datasets.py#L12-L22

https://github.com/BirkhoffG/cfnet/blob/24783713dc787cd5b13e70aa483e455c4856198f/cfnet/datasets.py#L35-L51

Expected Functionalities

NumpyDataset should contain all the input data.

# x, y are jax.numpy.array, such that len(x) == len(y)
dataset = NumpyDataset(x, y)

x, y = dataset[:] # access all the data of x, y
x_5, y_5 = dataset[:5] # access first five data of x, y

NumpyLoader iterates the NumpyDataset. See Pytorch Docs.

batch_size = 128
dataloader = NumpyLoader(
    dataset, # a `NumpyDataset`
    batchsize=batch_size,
    shuffle=True, # if True, shuffle the data; else, return the data in order
    drop_last=False # if True, discard the last batch (if len(dataset) % batchsize != 0); else, return the last batch
)

for x, y in dataloader:
    assert len(x) == batch_size
    assert len(y) == batch_size
    ...

Check monitor_metrics before actually finding the metric in logs.

Before this line:
https://github.com/BirkhoffG/cfnet/blob/2ee1a3203a9935e89b2ed8adf175ee1150fd9960/cfnet/_ckpt_manager.py#L54

Check monitor_metrics.

raise ValueError(...)

if monitor_metrics is not appropriately configured.

Move

• binary_cross_entropy in cfnet.methods.vanilla
• grad_update, cat_normalize in cfnet.training_module

into cfnet.utils

https://arxiv.org/abs/1907.02584

Plugin to #50

Reference:
https://stackoverflow.com/a/68293931

For example, remove this function:

1. Pass seed and batch_size to TabularDataModule.train_dataloader, TabularDataModule.val_dataloader, and TabularDataModule.test_dataloader.

2. batch_size should also be an argument in TrainingConfigs
   https://github.com/BirkhoffG/cfnet/blob/2ee1a3203a9935e89b2ed8adf175ee1150fd9960/cfnet/train.py#L15

3. Deprecated batch_size in DataConfigs

4. Finally, pass appropriate arguments:
   https://github.com/BirkhoffG/cfnet/blob/2ee1a3203a9935e89b2ed8adf175ee1150fd9960/cfnet/train.py#L58-L59

https://arxiv.org/abs/1905.07697

We use cat_normalize for encoding features, and clip continuous features to [0, 1]. This is because we use one-hot encoding for cat features, and min-max scalar for cont features.

If a user wants to use other encoding methods (e.g., standardized cont features), our current way of handing normalized data is not applicable.

Proposed features:

• #74
• #73

• #85
  

  ReLax/relax/data/module.py

  Line 404 in 729cacf

  def load_data(

• #86
• #87

We might want to provide a hook to the data_module for accessing cat_idx and cat_array.

https://github.com/BirkhoffG/cfnet/blob/6d5c3ec28351de79e575bbded5e67ab1bab48b03/cfnet/datasets.py#L95

You can write something like:

 batch_data = self.dataset[self.indices] 

Proposal

data_module = TabularDataModule('adult')

As such, TabularDataModule will automatically load data_configs of the adult dataset.

We should also allow TabularDataModule to pass user-defined configs (i.e., current argument data_configs: str | dict).

We don't support PredictiveTrainingModuleConfigs

https://github.com/NVIDIA-Merlin/dataloader

Plugin to #50

Create a Jupyter Notebook to benchmark methods in Carla