Comments (17)
weicheng113
commented on August 19, 2024
2
Hi @Zetmas , yes, you just drop the last batch, as the last batch may not be a full batch and there is a chance that it only contain 1 sample. I used pytorch dataloader to drop last batch, but you can somehow manually drop the last one.
from tpc-los-prediction.
weicheng113
commented on August 19, 2024
1
Hi Emma,
Just let you know the above error was because I had different batch_size(bigger batch_size) and the last batch may only contains one sample, which cannot perform batch normalization. I have fixed it.
~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/nn/functional.py in batch_norm(input, running_mean, running_var, weight, bias, training, momentum, eps)
2278 )
2279 if training:
-> 2280 _verify_batch_size(input.size())
2281
2282 return torch.batch_norm(
~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/nn/functional.py in _verify_batch_size(size)
2246 size_prods *= size[i + 2]
2247 if size_prods == 1:
-> 2248 raise ValueError("Expected more than 1 value per channel when training, got input size {}".format(size))
2249
2250
ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 64])I also modified the data generator to use pytorch dataloader so as to separate the data preprocessing into different process, which also allows multiple workers for data preprocessing and makes better GPU utilization. In this way, I reduced the eICU training for 15 epochs best TPC to less than 5 hours now.
from tpc-los-prediction.
EmmaRocheteau
commented on August 19, 2024
1
Yes in theory the transformer should have all the tools to do very well! And I think in the limit of infinite data it would outperform TPC because there is theoretically more capacity in the model. However specifically in the niche of big (but limited) EHR data the TPC model does very well :) Looking forward to seeing your results in the coming weeks!
from tpc-los-prediction.
EmmaRocheteau
commented on August 19, 2024
Hello! Thank you for your question :) I've just checked a couple of my experiment logs and for the best_tpc model for the LoS task alone on eICU it took just under 12 hours to train. This was on one NVIDIA TITAN Xp GPU. I think the reason why it fluctuates on memory use is things get transferred to the CPU from time to time for metric processing etc. but my memory is a little hazy. It is very possible (probable in fact!) that it is not the most efficient implementation it could be in terms of the computation time! I really hope that's helpful. Let me know if there is anything else I can do to help you
from tpc-los-prediction.
weicheng113
commented on August 19, 2024
Thanks Emma for the quick response. I am going to do a detailed learning on your source code. I can see there is a lot of comments, which are very helpful for me. So far it has been quite smooth for my learning.
The following training is using ml.g4dn.xlarge with 16GB. For 14 epoches, I guess it took less than 8 hours. But I got error for final epoch below. Do you know the reason for the error(I added some comments to tpc_model.py, so the line number will not be the same as your original code)? If not, I will try to debug myself. Thanks.
Experiment started.
Done epoch 0
Done epoch 1
Done epoch 2
Done epoch 3
Done epoch 5
Done epoch 7
Done epoch 8
Done epoch 9
Done epoch 10
Done epoch 11
Done epoch 13
Done epoch 14
File "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/trixi/experiment/experiment.py", line 90, in run
self.validate(epoch=self._epoch_idx)
File "/home/ec2-user/SageMaker/DL4H-Project/TPC-LoS-prediction/models/experiment_template.py", line 221, in validate
self.test()
File "/home/ec2-user/SageMaker/DL4H-Project/TPC-LoS-prediction/models/experiment_template.py", line 254, in test
y_hat_los, y_hat_mort = self.model(padded, diagnoses, flat)
File "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/ec2-user/SageMaker/DL4H-Project/TPC-LoS-prediction/models/tpc_model.py", line 626, in forward
diagnoses_enc = self.relu(self.main_dropout(self.bn_diagnosis_encoder(self.diagnosis_encoder(diagnoses)))) # B * diagnosis_size
File "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/ec2-user/SageMaker/DL4H-Project/TPC-LoS-prediction/models/tpc_model.py", line 78, in forward
training=True, momentum=exponential_average_factor, eps=self.eps) # set training to True so it calculates the norm of the batch
File "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/nn/functional.py", line 2280, in batch_norm
_verify_batch_size(input.size())
File "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/nn/functional.py", line 2248, in _verify_batch_size
raise ValueError("Expected more than 1 value per channel when training, got input size {}".format(size))
ValueError('Expected more than 1 value per channel when training, got input size torch.Size([1, 64])',)
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-6-f696031b3a88> in <module>
----> 1 run_best_tpc()
~/SageMaker/DL4H-Project/TPC-LoS-prediction/main_tpc.py in run_best_tpc()
52 base_dir=log_folder_path,
53 explogger_kwargs={'folder_format': '%Y-%m-%d_%H%M%S{run_number}'})
---> 54 tpc.run()
55
56
~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/trixi/experiment/experiment.py in run(self, setup)
106 self._exp_state = "Error"
107 self._time_end = time.strftime("%y-%m-%d_%H:%M:%S", time.localtime(time.time()))
--> 108 self.process_err(e)
109
110 def run_test(self, setup=True):
~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/trixi/experiment/pytorchexperiment.py in process_err(self, e)
389 self.elog.text_logger.log_to("\n".join(traceback.format_tb(e.__traceback__)), "err")
390 self.elog.text_logger.log_to(repr(e), "err")
--> 391 raise e
392
393 def update_attributes(self, var_dict, ignore=()):
~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/trixi/experiment/experiment.py in run(self, setup)
88 while self._epoch_idx < self.n_epochs and not self.__stop:
89 self.train(epoch=self._epoch_idx)
---> 90 self.validate(epoch=self._epoch_idx)
91 self._end_epoch_internal(epoch=self._epoch_idx)
92 self._epoch_idx += 1
~/SageMaker/DL4H-Project/TPC-LoS-prediction/models/experiment_template.py in validate(self, epoch, mort_pred_time)
219
220 elif self.config.mode == 'test' and epoch == self.n_epochs - 1:
--> 221 self.test()
222
223 if epoch == self.n_epochs - 1 and self.config.save_results_csv:
~/SageMaker/DL4H-Project/TPC-LoS-prediction/models/experiment_template.py in test(self, mort_pred_time)
252 padded, mask, diagnoses, flat, los_labels, mort_labels, seq_lengths = batch
253
--> 254 y_hat_los, y_hat_mort = self.model(padded, diagnoses, flat)
255 loss = self.model.loss(y_hat_los, y_hat_mort, los_labels, mort_labels, mask, seq_lengths, self.device,
256 self.config.sum_losses, self.config.loss)
~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
1100 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
1101 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1102 return forward_call(*input, **kwargs)
1103 # Do not call functions when jit is used
1104 full_backward_hooks, non_full_backward_hooks = [], []
~/SageMaker/DL4H-Project/TPC-LoS-prediction/models/tpc_model.py in forward(self, X, diagnoses, flat, time_before_pred)
624 next_X[:, :, time_before_pred:].permute(0, 2, 1).contiguous().view(B * (T - time_before_pred), -1)), dim=1) # (B * (T - time_before_pred)) * (((F + Zt) * (1 + Y)) + no_flat_features) for tpc
625 else:
--> 626 diagnoses_enc = self.relu(self.main_dropout(self.bn_diagnosis_encoder(self.diagnosis_encoder(diagnoses)))) # B * diagnosis_size
627 combined_features = cat((flat.repeat_interleave(T - time_before_pred, dim=0), # (B * (T - time_before_pred)) * no_flat_features
628 diagnoses_enc.repeat_interleave(T - time_before_pred, dim=0), # (B * (T - time_before_pred)) * diagnosis_size
~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
1100 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
1101 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1102 return forward_call(*input, **kwargs)
1103 # Do not call functions when jit is used
1104 full_backward_hooks, non_full_backward_hooks = [], []
~/SageMaker/DL4H-Project/TPC-LoS-prediction/models/tpc_model.py in forward(self, input)
76 return F.batch_norm(
77 input, self.running_mean, self.running_var, self.weight, self.bias,
---> 78 training=True, momentum=exponential_average_factor, eps=self.eps) # set training to True so it calculates the norm of the batch
79
80
~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/nn/functional.py in batch_norm(input, running_mean, running_var, weight, bias, training, momentum, eps)
2278 )
2279 if training:
-> 2280 _verify_batch_size(input.size())
2281
2282 return torch.batch_norm(
~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/nn/functional.py in _verify_batch_size(size)
2246 size_prods *= size[i + 2]
2247 if size_prods == 1:
-> 2248 raise ValueError("Expected more than 1 value per channel when training, got input size {}".format(size))
2249
2250
ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 64])from tpc-los-prediction.
weicheng113
commented on August 19, 2024
By the way, I guess the experiment does not support multiple GPU, right? as I don't see distributed training related code or the use of pytorch-lightning. Thanks.
from tpc-los-prediction.
EmmaRocheteau
commented on August 19, 2024
Hi! Yeah sorry I didn't work multiple GPUs into the code. I was mainly focused on proving whether the architecture was useful :') hopefully with reliable enough code to make it through however many runs I needed to get the results. I'm sorry I'm not familiar with that error, although I feel like I've seen something similar while I've worked on other projects since. Is the version of pytorch the same as the one I used (1.5)?
Also note that you've trained 15 epochs not 14, since python has weird indexing throughout!
from tpc-los-prediction.
weicheng113
commented on August 19, 2024
Thanks a lot for your time and reply. It is possible that I used a different version of pytorch, as I was trying to use latest version of libraries. I will have a look.
from tpc-los-prediction.
weicheng113
commented on August 19, 2024
By the way, the reason of using groups in 1D convoluation is to group convolving a field and its corresponding mask field together, right? This is my first time seeing the use of groups, which looks interesting. Thanks.
from tpc-los-prediction.
EmmaRocheteau
commented on August 19, 2024
Ah brilliant! If someone else has this issue I will be sure to let them know. In response to your second question - yes, it's to pair the variables with their corresponding mask. It took quite a bit of fiddling about with and checking to ensure the pairing was correct. Thank you for your interest in the code and the project. I'm very happy you are using it!
from tpc-los-prediction.
weicheng113
commented on August 19, 2024
Thanks for your confirmation and explanation, Emma. It is very interesting that the model performs better than transformer. I will do a detailed study in the following weeks.
from tpc-los-prediction.
weicheng113
commented on August 19, 2024
Hi Emma,
Sorry to bother you. Did you get trixi browser to work? I got the following error and I try to find information on trixi github issues, but no one mentioned it before.
File "C:\Users\weich\Anaconda3\envs\dl4h-project\lib\site-packages\flask\templating.py", line 149, in render_template
ctx.app.jinja_env.get_or_select_template(template_name_or_list),
File "C:\Users\weich\Anaconda3\envs\dl4h-project\lib\site-packages\jinja2\environment.py", line 1071, in get_or_select_template
return self.get_template(template_name_or_list, parent, globals)
File "C:\Users\weich\Anaconda3\envs\dl4h-project\lib\site-packages\jinja2\environment.py", line 1000, in get_template
return self._load_template(name, globals)
File "C:\Users\weich\Anaconda3\envs\dl4h-project\lib\site-packages\jinja2\environment.py", line 959, in _load_template
template = self.loader.load(self, name, self.make_globals(globals))
File "C:\Users\weich\Anaconda3\envs\dl4h-project\lib\site-packages\jinja2\loaders.py", line 126, in load
source, filename, uptodate = self.get_source(environment, name)
File "C:\Users\weich\Anaconda3\envs\dl4h-project\lib\site-packages\flask\templating.py", line 59, in get_source
return self._get_source_fast(environment, template)
File "C:\Users\weich\Anaconda3\envs\dl4h-project\lib\site-packages\flask\templating.py", line 95, in _get_source_fast
raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: overview.html
127.0.0.1 - - [17/Apr/2022 11:11:09] "GET / HTTP/1.1" 500 -I tried to look into trixi related source code. It seems it requires some html webpages(e.g., overview.html), but how to get these html files?
# trixi.experiment_browser.browser.py
def register_url_routes(app, base_dir):
app.add_url_rule("/", "overview", lambda: overview(base_dir), methods=["GET"])
app.add_url_rule("/overview", "overview_", lambda: overview_(base_dir), methods=["GET"])
app.add_url_rule('/experiment', "experiment", lambda: experiment(base_dir), methods=['GET'])
app.add_url_rule('/combine', "combine", lambda: combine(base_dir), methods=['GET'])
app.add_url_rule('/experiment_log', "experiment_log", lambda: experiment_log(base_dir), methods=['GET'])
app.add_url_rule('/experiment_plots', "experiment_plots", lambda: experiment_plots(base_dir), methods=['GET'])
app.add_url_rule('/experiment_remove', "experiment_remove", lambda: experiment_remove(base_dir), methods=['GET'])
app.add_url_rule('/experiment_star', "experiment_star", lambda: experiment_star(base_dir), methods=['GET'])
app.add_url_rule('/experiment_rename', "experiment_rename", lambda: experiment_rename(base_dir), methods=['GET'])
def overview(base_dir):
try:
base_info = process_base_dir(base_dir, ignore_keys=IGNORE_KEYS)
base_info["title"] = base_dir
return render_template("overview.html", **base_info)
except Exception as e:
print(e.__repr__())
raise e
abort(500)Thanks
from tpc-los-prediction.
EmmaRocheteau
commented on August 19, 2024
Sorry for the delay! There was a long weekend holiday in the UK and I went away. I checked my trixi installation and I definitely have html files there. I don't know if this is helpful but I will attach my trixi:
trixi.zip
You can find them in experiment browser/templates/
from tpc-los-prediction.
weicheng113
commented on August 19, 2024
Thank you very much, Emma. Ok, I think I know the problem. I can now see the html files sitting under templates. I modified trixi browser to work in microsoft windows machine, which is why it can't not find these template html files. I think trixi expect linux and mac os users. I had to modify line 68 in browser.py to make it work in wondows.
def create_flask_app(base_dir):
...
blueprint = Blueprint("data", __name__, static_url_path='/' + base_dir, static_folder=base_dir)
# original code below:
# blueprint = Blueprint("data", __name__, static_url_path=base_dir, static_folder=base_dir)
...With your hint, I think I should somehow be able to make it work. Thanks.
from tpc-los-prediction.
EmmaRocheteau
commented on August 19, 2024
Sorry about all the teething issues you are having! I will try to be very responsive to any further issues
from tpc-los-prediction.
weicheng113
commented on August 19, 2024
Thanks Emma. Appreciate your time. There is a lot to learn from your paper. You can reply when you have time.
from tpc-los-prediction.
Zetmas
commented on August 19, 2024
Just let you know the above error was because I had different batch_size(bigger batch_size) and the last batch may only contains one sample, which cannot perform batch normalization. I have fixed it.
Hello Cheng @weicheng113 , would you mind sharing how you fix this problem? I met the same batch size issue. Thanks!
from tpc-los-prediction.
Related Issues (17)
- performance HOT 3
- Preproceesing eICU issue HOT 3
- Preprocessing MIMIC-IV issue HOT 1
- raise DataError('No numeric types to aggregate') pandas.core.base.DataError: No numeric types to aggregate HOT 2
- File "pandas\_libs\parsers.pyx", line 545, in pandas._libs.parsers.TextReader.__cinit__ pandas.errors.EmptyDataError: No columns to parse from file HOT 5
- AttributeError: 'Config' object has no attribute 'model_type' HOT 4
- Issues preprocessing MIMIC-IV using BigQuery HOT 1
- n_epochs does not work as an argument. HOT 1
- Channel mismatch in model_type 'tpc' HOT 1
- KeyError: '[141939 142056 142476 142521 142560 146391 147447 149039 149606 153006\n 160529 162431 166572 166709 167391 167417 171174 175528 177651 178069\n 178858 179142 179554] not in index' HOT 2
- Computation Power needed
- kappa for los HOT 2
- Python Version HOT 2
- some conceptual questions about temp_pointwise HOT 9
- Some questions about performance HOT 19
- Question regarding masked datafields in timeseries.csv processed file HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tpc-los-prediction.