The deepcollisionalcrosssection from mannlabs

deepcollisionalcrosssection's People

Contributors

Stargazers

Watchers

deepcollisionalcrosssection's Issues

Cannot create a tensor proto whose content is larger than 2GB

This occurs when running predict.py on large files. I was using a 10,000,000 peptide input file.

I fixed this issue by creating a generator to load the files in batches.

def get_tf_dataset(one_dat, lab, meta_data, data, model_params):
    # XXX reverse the sequences for testing of attention, could probably be done somewhere else, but here it is easy
    reverse = False
    # attention_mask = (one_dat != 22)
    # print(data['lens'].values)
    # attention_mask[:,:,data['lens'].values] = True
    # print(attention_mask)

    if reverse:
        for i, row in enumerate(one_dat):
            no_dum = row[ row != 22]
            no_dum_flip = np.flip(no_dum,0)
            one_dat[i,:len(no_dum)] = no_dum_flip[:, np.newaxis]

    print("preparing data")
    data_chunks = list(np.array_split(one_dat, model_params['batch_size']))
    labels_chunks = list(np.array_split(lab, model_params['batch_size']))
    dm_chunks = list(np.array_split(meta_data, model_params['batch_size']))
    l_chunks = list(np.array_split(data['lens'].values, model_params['batch_size']))
    dtask_chunks = list(np.array_split(data['task'].values, model_params['batch_size']))

    def genenerator():
        for i, j, k, l, m in zip(data_chunks, dm_chunks, labels_chunks, l_chunks, dtask_chunks):
            yield i, j, k, l, m

    print("preparing batched datset")
    dataset_test = tf.data.Dataset.from_generator(genenerator, (tf.float32, tf.float32, tf.float32, tf.float32, tf.float32))
    dataset_test = dataset_test.apply(tf.contrib.data.unbatch())
    dataset_test = dataset_test.batch(model_params['batch_size'])
    dataset_test = dataset_test.prefetch(1) 

    return dataset_test

References:
https://stackoverflow.com/questions/55122902/from-tensor-slices-with-big-numpy-array-while-using-tf-keras
https://stackoverflow.com/questions/49531286/tensorflow-tf-data-dataset-cannot-batch-tensors-with-different-shapes-in-compo

Recommend Projects