alvinwan / neural-backed-decision-trees Goto Github PK
View Code? Open in Web Editor NEWMaking decision trees competitive with neural networks on CIFAR10, CIFAR100, TinyImagenet200, Imagenet
Home Page: https://nbdt.aaalv.in
License: MIT License
Making decision trees competitive with neural networks on CIFAR10, CIFAR100, TinyImagenet200, Imagenet
Home Page: https://nbdt.aaalv.in
License: MIT License
I mean apply NBDT with soft loss.., you know the inner node weight can't be update with the soft loss in binary classfication due to the special inner probability computing method.
Is there any example for a regression problem?
Hi,
Great project and article, congratulations.
I'm wondering if, once the training is finished, the final NBDT can be fully exported as a JIT file?
It seems that everything is written in pytorch and inference can be done in a single forward pass so it should be fine but I'm still a bit worried about the tree part and the self.rules.forward_with_decisions(x)
. Have you ever tried to export one of your models as JIT? Does it work as is?
Thanks in advance for your answer.
Can you answer how Zero-Shot Superclass Generalization is calculated?
when run command 'nbdt-hierarchy' in ubuntu 18.04
nbdt-hierarchy --checkpoint=wrn28_10_cifar10 --dataset=CIFAR10
nbdt-hierarchy --vis-sublabels --vis-zoom=1.25 --dataset=CIFAR10 --arch=wrn28_10_cifar10
Here is ModuleNotFoundError: No module named 'nbdt.thirdparty',
I wonder what happened
Hey,
Thanks for sharing your code.
I have a pretrained pytorch(Fastai) model on my custom dataset and when i am trying to run the below code on my dataset. I get an assertion error.
The dataset I have is loaded using a dataloader having 3 classes
from nbdt.loss import SoftTreeSupLoss
criterion = nn.CrossEntropyLoss()
criterion = SoftTreeSupLoss(dataset='CIFAR10', criterion=criterion) -- originial
criterion = SoftTreeSupLoss(dataset=data, criterion=criterion) -- custom
Also, do I have to train the whole model again or can just pass this new loss function to that model and get the inference?
Any help would be really appreciated :)
Hello, I am trying to use softTreeLoss by using following codes:
from nbdt.loss import SoftTreeSupLoss
train_loss_fn = nn.CrossEntropyLoss().cuda()
criterion = SoftTreeSupLoss(criterion=train_loss_fn, dataset='Imagenet1000', tree_supervision_weight=1.0,
hierarchy='induced-efficientnet_b7b')
...
for i, (input, targets) in enumerate(train_loader):
targets = targets.cuda(async=True)
input_var = torch.autograd.Variable(input).cuda()
targets_var = torch.autograd.Variable(targets).cuda()
scores = model(input_var)
loss = criterion(scores, targets_var)
Then it comes the following errors:
File "/gruntdata/semantic-hierarchy-master/neural-backed-decision-trees/nbdt/model.py", line 240, in forward
wnid_to_outputs = self.forward_nodes(outputs)
File "/gruntdata/semantic-hierarchy-master/neural-backed-decision-trees/nbdt/model.py", line 101, in forward_nodes
return self.get_all_node_outputs(outputs, self.nodes)
File "/gruntdata/semantic-hierarchy-master/neural-backed-decision-trees/nbdt/model.py", line 90, in get_all_node_outputs
node_logits = cls.get_node_logits(outputs, node)
File "/gruntdata/semantic-hierarchy-master/neural-backed-decision-trees/nbdt/model.py", line 79, in get_node_logits
for new_label in range(node.num_classes)
File "/gruntdata/semantic-hierarchy-master/neural-backed-decision-trees/nbdt/model.py", line 79, in
for new_label in range(node.num_classes)
AttributeError: 'Tensor' object has no attribute 'T'
When I cloned the repository and ran main.py (without changing anything) I get this output:
not enough values to unpack (expected 2, got 0)
==> Preparing data..
Files already downloaded and verified
Files already downloaded and verified
Training with dataset CIFAR10 and 10 classes
==> Building model..
==> Checkpoints will be saved to: ./checkpoint/ckpt-CIFAR10-ResNet18.pth
classes: (callable)
Epoch: 0
Traceback (most recent call last):
File "<ipython-input-7-f9bd8031870b>", line 1, in <module>
runfile('C:/Users/Matthew Chen/Documents/GitHub/neural-backed-decision-trees/main.py', wdir='C:/Users/Matthew Chen/Documents/GitHub/neural-backed-decision-trees')
File "C:\Users\Matthew Chen\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 827, in runfile
execfile(filename, namespace)
File "C:\Users\Matthew Chen\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 110, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)
File "C:/Users/Matthew Chen/Documents/GitHub/neural-backed-decision-trees/main.py", line 315, in <module>
train(epoch, analyzer)
File "C:/Users/Matthew Chen/Documents/GitHub/neural-backed-decision-trees/main.py", line 227, in train
for batch_idx, (inputs, targets) in enumerate(trainloader):
File "C:\Users\Matthew Chen\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 279, in __iter__
return _MultiProcessingDataLoaderIter(self)
File "C:\Users\Matthew Chen\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 719, in __init__
w.start()
File "C:\Users\Matthew Chen\Anaconda3\lib\multiprocessing\process.py", line 112, in start
self._popen = self._Popen(self)
File "C:\Users\Matthew Chen\Anaconda3\lib\multiprocessing\context.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\Matthew Chen\Anaconda3\lib\multiprocessing\context.py", line 322, in _Popen
return Popen(process_obj)
File "C:\Users\Matthew Chen\Anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 89, in __init__
reduction.dump(process_obj, to_child)
File "C:\Users\Matthew Chen\Anaconda3\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
BrokenPipeError: [Errno 32] Broken pipe
I'm not sure if I downloaded all the packages correctly, but this seems to be an error where the request to some server is blocked/timed out?
Can I find documentation somewhere?
I've foudn the colab notebook useful for the single prediction, but I can't get the different labels of a forward with decision tree.
Just wondering what is the loop "for _loss in args.loss:" in line 195 of main.py for? It seems like the loss is always overridden by the latest one.
Can it run under Windows system, and how?
This is pretty cool, I was just wondering if it also works on continuous data. I think resnet can work on continuous data, so I'm curious.
Hi, I'm trying to use gen_train_eval_nopretrained.sh to train with a new dataset I implemented. However, in main.py there is this line of code
tree = Tree.create_from_args(args, classes=trainset.classes)
the error I got is FileNotFound at nbdt/hierarchies/mydataset/graph-induced.json
and it seems that this line of code requires generated hierarchy? But I can only generate hierarchy after I've trained the model. So I'm a bit confused on what to do.
it has different outputs when I input the same picture into the model which is trained by myself.However when use the pretrained '.pth' model,the output is right,and the probability tensor is ensure,in contrast,my probability tensor will change in every tests.
I trained model use python main.py --lr=0.01 --dataset=CIFAR10 --arch=wrn28_10_cifar10 --hierarchy=induced-wrn28_10_cifar10 --pretrained --loss=SoftTreeSupLoss.
input is dog.jpg,twice outputs are below
out tensor([[0.1268, 0.1282, 0.1303, 0.0642, 0.0618, 0.0623, 0.1216, 0.0618, 0.1176,
0.1254]], grad_fn=)
result
bird
out tensor([[0.1245, 0.1192, 0.1246, 0.0643, 0.0636, 0.0649, 0.1203, 0.0634, 0.1291,
0.1261]], grad_fn=)
result
ship
Hi NBDT,
When I ran the code below....
from nbdt.model import SoftNBDT
from nbdt.models import ResNet18, wrn28_10_cifar10, wrn28_10_cifar100, wrn28_10 # use wrn28_10 for TinyImagenet200
model = wrn28_10_cifar10()
model = SoftNBDT(
pretrained=True,
dataset='CIFAR10',
arch='wrn28_10_cifar10',
model=model)
I received the following error
not enough values to unpack (expected 2, got 0)
Downloading: "https://github.com/alvinwan/neural-backed-decision-trees/releases/download/0.0.1/ckpt-CIFAR10-wrn28_10_cifar10-induced-wrn28_10_cifar10-SoftTreeSupLoss.pth" to /home/jupyter/.cache/torch/checkpoints/ckpt-CIFAR10-wrn28_10_cifar10-induced-wrn28_10_cifar10-SoftTreeSupLoss.pth
Did it work?
Thank you
Vivek
Hello @alvinwan and team!
First off, thank you for the awsome work. As per this repo and your paper, I noticed that the applications of the NBDT technique have been limited to images. Do you foresee this technique being used in other domains such as NLP?
I am interested in working on an interpretable decision-tree learning framework which can come close to DNNs' performance in NLP, and am drawing some inspiration from your work. Would be great to get your thoughts on this :)
Hello,
I am trying to integrate the rare planes dataset with this code repository. I followed the steps for a custom dataset in the README and I was able to get one of the scripts running (partially). I can run 'step 0' and 'step 1' of the script gen_train_eval_nopretrained.sh, however, when I run 'step 2' I get a CUDA out of memory error. Any ideas on why this would happen?
Note: I have tried lowering the batch size but that does not seem to affect the error message.
For reference here is the script I am running:
# Want to train with wordnet hierarchy? Just set '--hierarchy=wordnet' below.
# This script is for networks that do NOT come with a pretrained checkpoint provided either by a model zoo or by the NBDT utility itself.
model="ResNet18"
dataset=RarePlanes
weight=1
batch_size=4
# 0. train the baseline neural network
python main.py --dataset=${dataset} --arch=${model} --batch-size=${batch_size}
# 1. generate hieararchy -- for models without a pretrained checkpoint, use 'checkpoint'
nbdt-hierarchy --dataset=${dataset} --checkpoint=./checkpoint/ckpt-${dataset}-${model}.pth
# 2. train with soft tree supervision loss -- for models without a pretrained checkpoint, use 'path-resume' OR just train from scratch, without 'path-resume'
# python main.py --lr=0.01 --dataset=${dataset} --model=${model} --hierarchy=induced-${model} --path-resume=./checkpoint/ckpt-${dataset}-${model}.pth --loss=SoftTreeSupLoss --tree-supervision-weight=${weight} # fine-tuning
python main.py --dataset=${dataset} --arch=${model} --hierarchy=induced-${model} --loss=SoftTreeSupLoss --tree-supervision-weight=${weight} # training from scratch
# 3. evaluate with soft then hard inference
for analysis in SoftEmbeddedDecisionRules HardEmbeddedDecisionRules; do
python main.py --dataset=${dataset} --arch=${model} --hierarchy=induced-${model} --loss=SoftTreeSupLoss --eval --resume --analysis=${analysis} --tree-supervision-weight=${weight}
done
And here is the error I get on line 17:
Training with dataset RarePlanes and 54 classes
==> Building model..
==> Checkpoints will be saved to: ./checkpoint/ckpt-RarePlanes-ResNet18-induced-ResNet18-SoftTreeSupLoss.pth
path_graph: /home/pguerrie/neural-backed-decision-trees/nbdt/hierarchies/RarePlanes/graph-induced-ResNet18.json
path_wnids: /home/pguerrie/neural-backed-decision-trees/nbdt/wnids/RarePlanes.txt
tree_supervision_weight: 1.0
classes: (callable)
dataset: (callable)
criterion: (callable)
classes: (callable)
Epoch: 0
Traceback (most recent call last):
File "main.py", line 315, in <module>
train(epoch, analyzer)
File "main.py", line 230, in train
outputs = net(inputs)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
result = self.forward(*input, **kwargs)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
return self.module(*inputs[0], **kwargs[0])
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
result = self.forward(*input, **kwargs)
File "/home/pguerrie/neural-backed-decision-trees/nbdt/models/resnet.py", line 112, in forward
out = self.features(x)
File "/home/pguerrie/neural-backed-decision-trees/nbdt/models/resnet.py", line 102, in features
out = F.relu(self.bn1(self.conv1(x)))
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
result = self.forward(*input, **kwargs)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py", line 107, in forward
exponential_average_factor, self.eps)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/functional.py", line 1670, in batch_norm
training, momentum, eps, torch.backends.cudnn.enabled
RuntimeError: CUDA out of memory. Tried to allocate 6.12 GiB (GPU 0; 11.91 GiB total capacity; 6.45 GiB already allocated; 4.86 GiB free; 6.47 GiB reserved in total by PyTorch)
I don't understand why I would be able to run step 0 with a large batch size but I can't run step 2 even with a very small batch size. I was thinking the networks were largely the same (except for the extra output nodes in step 2 due to the hierarchy and the hierarchical loss used to train). Any help would be greatly appreciated!
Hi, I wonder if there are any mlp based implementation for NBDT,which can be used in areas other than CV. Thanks a lot!
Hello, the current code setting in ImageNet, I can only feed batch-size=16 one gpu for resnet50 model with ImageNet database, my gpu is V100, however, the same settings in my friends computer the batch-size=36.
nbdt /imaginary/path/to/local/image.png`
is this not available yet?
According to the paper at
3.1 Inference with Embedded Decision Rules
First, our NBDT approach featurizes each sample using the neural network
backbone; the backbone consists of all neural network layers before the final
fully-connected layer.
So, It means the sample is run on the NN architecture excluding the final fully-connected layer(?)
If so, Why are you including the final fully-connected layer in here before passing x
to self.rules.forward_with_decisions
?
neural-backed-decision-trees/nbdt/model.py
Lines 323 to 324 in 7ef5fe5
Can you please explain?
Has this paper been published? I cannot find the information.
When I call model.forward_with_decisions to get the intermediate decisions on Colab, we get a list of 'node': <nbdt.data.custom.Node object at 0x7f2318a35a90> objects. How could I get access to the information within the node? And how could I generate a visualization of a decision tree based on this result?
Hey,
This looks like a great project,
Does it support or will support Tensorflow engine as well? ?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.