alexanderkroner / saliency Goto Github PK

Thank you Sir for the wonderful code
I have a question，when I test on MIT300， how do I get the the log density predictions of a probability model

This code is not executing on TF V 2.4.1

Is it possible to run the code to video clip, not a through javascript but using Python?
Does the video example use specialized model or adopt same pipeline to frame by frame data?

Hi, I'd just like to double-check that I'm doing what I think I'm doing. If I'd like to reproduce the results of your paper on my dataset, I need to use: python main.py test -d mit1003 -p [path to my images] - this will download/use a model first trained on SALICON and fine-tuned on MIT1003, which is what you presented in the paper. Correct? Thanks in advance!

First, it's a great job.

I have this issue each time I process an image. The memory is rising till filling out, I tried to use session.close() or tf.keras.backend.clear_session but I don't have results. Is there any way to stop the rising o refresh the memory each time process the algorithm?

Thank you in advance

Just wanted to leave a note in case anyone else runs into this issue.
After upgrading to an RTX 3090, I was getting bad outputs with Tensorflow 1.13. Tensorflow 1.15 causes the program to hang.
Switching to nvidia-tensorflow gets back the great results from the model.

Hi

I'm trying to test your pretrained model on SALICON with my images. I used #15 this method to convert tf1 model to tf2 however, output seems not correct to me.

Here is my code,

import tensorflow as tf

imported = tf.saved_model.load("./converted_model")
imported = imported.signatures["serving_default"]

img = tf.io.read_file("face.jpg")
tensor = tf.io.decode_image(img, channels=3, dtype=tf.dtypes.float32)
#tensor = tf.image.resize(tensor, [240, 320])
input_tensor = tf.expand_dims(tensor, axis=0)
# input_tensor = tf.ones((1, 240, 320, 3))
saliency = imported(input_tensor)["output"]

tf.keras.preprocessing.image.save_img('face_out.jpg',saliency[0])

Here is the input image

Here is the output image

Thanks!

TF 1.13 is quite outdated. TF 2.11 is the latest and does not support graphs. Any chance of updating the code or helping out on how this can work with newer TF versions?

Dear authors,

I am trying to fine-tune the provided '.pd' format pretrained model but failed.
May I request a re-trainable '.ckpt' format pretrained model on SALICON?

Thanks!

Hi Alexander,
I'm trying to use your pre-trained weights using python ./saliency/main.py test -d mit1003 -p on google colab. This code worked last week but is now displaying the following error :

>> Downloading pre-trained weights...Access denied with the following error:

 	Cannot retrieve the public link of the file. You may need to change
	the permission to 'Anyone with the link', or have had many accesses. 

You may still be able to access the file from the browser:

	 https://drive.google.com/uc?id=1P_tWxBl3igZlzcHGp5H3T3kzsOskWeG6&export=download 

Traceback (most recent call last):
  File "./saliency/main.py", line 249, in <module>
    main()
  File "./saliency/main.py", line 245, in main
    test_model(args.data, paths, config.PARAMS["device"])
  File "./saliency/main.py", line 174, in test_model
    model_name[:-3])
  File "/content/saliency/download.py", line 469, in download_pretrained_weights
    with zipfile.ZipFile(data_path + "tmp.zip", "r") as zip_ref:
  File "/usr/lib/python3.7/zipfile.py", line 1240, in __init__
    self.fp = io.open(file, filemode)
FileNotFoundError: [Errno 2] No such file or directory: '/content/saliency/weights/tmp.zip'```

hi，the link of SALICON dataset is invalid
could you pleases offer new link?

Hi, thank you Sir for the wonderful code you have made here.

I want to test myself dataset. Because My TF V was 2.4.1, I have seen the previous issue. But I did not know how to put myself data to test.

I am trying to convert the model to TFLite, I ran into some problems. How did you convert it to Tensorflow.js? If I could get some pointers on how to convert it to JS I could probably get it to convert to Lite as well.

I download the dataset too slowly with download.py script. Even it's interrupted. Could you please upload these datasets in Google driver to Baidu Cloud, which is available faster for people in china.

Thanks in advance.

Thank you Sir for the wonderful code you have made here... thoroughly worked and clearly coded..

Sir, i tried to train on salicon and tested on (say) osie dataset. The predicted saliency maps are obtained on the results folder perfectly. I would like to know how i will be able to compute different saliency metric (say kld, AUC-Judd) between the predicted and true maps via code frame? How can I implement an evaluate function between them? It would be great if you can provide help towards this..

First of all, thanks for the detailed solution, readable and reusable code using tensorflow.
Is it possible to combine all datasets and train network on all available data?
I've read the README, and as far as I understood at first network must be trained on salicon, than executing train for the second time, it can fine tume the previous parameters for mit1003 or cat2000.

And is it possible to add to the list the dataset for webpages (dropbox: https://www.dropbox.com/s/30nxg2uwd1wpb80/webpage_dataset.zip?dl=0) mentioned here http://saliency.mit.edu/datasets.html (Fixations in Webpage Images (FiWI) data set)?

I have installed
python 3.6 with keras(2.3.1)
i tried uninstalling and reinstalling still I get the error

could u please help me in this

No code changes have been made, and the version of the library is compliant
This is the result of my training

Only the first picture was correct when the 10 pictures were tested

Thanks to reassure!

I am trying to run demo with a tensorflow version 1.2.1 (next version from current 1.1.2) but in browser gives following error [email protected]:2 Uncaught (in promise) Error: The shape of dict['input'] provided in model.execute(dict) must be [], but was [1,48,64,3].

Hi， Thank you for sharing the good idea and code. The code can be run in Python 3.5 and tensorflow 1.10.1 environments. At the same time, I have a few questions to ask.

1: How to get the results of 8 indicators in Table 1 and Table 2 on MIT300 test set and CAT2000 test set? I didn't see these eight indicators in the program.

2: How to visualize the image in the validation set? As shown in Figure 3 of the article. I don't seem to see the corresponding visualization programs or the results in color maps.

Can you give me some advice?

Thanks !

Hello, what is the frame rate of the saliency detection video you provide, and how is the quality？
thank u~

I wanted to use your method to determine the composition of a photo, but I don't know how to do it in real time with a web camera as in your demo.
If you don't mind, could you tell me how to do it with the distributed code?

(finalwork) C:\Users\86185\Desktop\saliency-master>python main.py test -d cat2000 -p C:\Users\86185\Desktop\saliency-master\tset

Downloading pre-trained weights...Traceback (most recent call last):
File "main.py", line 249, in
main()
File "main.py", line 245, in main
test_model(args.data, paths, config.PARAMS["device"])
File "main.py", line 174, in test_model
model_name[:-3])
File "C:\Users\86185\Desktop\saliency-master\download.py", line 469, in download_pretrained_weights
with zipfile.ZipFile(data_path + "tmp.zip", "r") as zip_ref:
File "C:\Users\86185\anaconda3\envs\finalwork\lib\zipfile.py", line 1131, in init
self._RealGetContents()
File "C:\Users\86185\anaconda3\envs\finalwork\lib\zipfile.py", line 1198, in _RealGetContents
raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file

did i set the wrong environment or miss some steps?

After successfully download salicon it was showing error "tensorflow.data has no attribute experimental
can you give me the solution?

Hi,
I am trying to run the test function with the GPU, but I am getting this error:
"Default MaxPoolingOp only supports NHWC on device type CPU"
Everything works well when I use the CPU, though.

Best regards,

Hi @alexanderkroner, thank you for such a nice piece of code and thorough documentation for the repository.

How can I train a model on a custom/new dataset? My dataset contains eye-tracking videos of 10 participants, ranging from 3 to 4 mins. I have the gaze information, fixations, and timestamps. From what I understand, I have to first prepare the dataset as SALICON dataset that is accepted and read by the code and then train the model. However, I have some confusion and would like to ask questions that will help me in the right direction.

1. Dataset preparation

• convert videos into frames such that individual frame contains the gaze data of one participant and its time stamp and fixation information.
• generate a map of the individual images and feed it as a ground truth.

2. Model training

• first train on the SALICON dataset to pre-train it.
• use the pre-train model to fine-tune my dataset.

To fine-tune the model, what parameters must I change, or how would I incorporate the SILICON pre-trained model in the command to read the pre-trained model and then later use it on my dataset?

Thank you so much for your time.

Hi alexander,
FIrst of all, I would like to appreciate your work and thanks for putting it over GITHUB.
while downloading the cat2000 dataset using "python main.py train -d cat2000" the model produces an error i.e.
ConnectionResetError: [Errno 104] Connection reset by peer. Although the dataset can be downloaded from the browser, however when using command i.e. "python main.py train -d cat2000" it produces an error. If you have any suggestion please share.
Thanks
Regards,
Wmir

Complete Error ( using Google Colab) :

Downloading CAT2000 dataset...>> Download done...
/content/saliency/data/
Traceback (most recent call last):
File "/content/saliency/main.py", line 249, in
main()
File "/content/saliency/main.py", line 243, in main
train_model(args.data, paths, config.PARAMS["device"])
File "/content/saliency/main.py", line 74, in train_model
iterator = data.get_dataset_iterator("train", dataset, paths["data"])
File "/content/saliency/data.py", line 483, in get_dataset_iterator
dataset_class = getattr(current_module, class_name)(data_path)
File "/content/saliency/data.py", line 160, in init
download.download_cat2000(parent_path)
File "/content/saliency/download.py", line 148, in download_cat2000
with zipfile.ZipFile(data_path + "tmp.zip", "r") as zip_ref:
File "/usr/lib/python3.7/urllib/request.py", line 276, in urlretrieve
block = fp.read(bs)
File "/usr/lib/python3.7/http/client.py", line 461, in read
n = self.readinto(b)
File "/usr/lib/python3.7/http/client.py", line 505, in readinto
n = self.fp.readinto(b)
File "/usr/lib/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer

Dear @alexanderkroner,

I'm quite interested in your great project on Saliency. Would you be able to provide some instructions for me on how to run '.py' code files within the '.ipynb' notebooks in Jupyter or Google Colab you've provided?

Additionally, I'm curious if there's flexibility to build my own model using your work as a base or if I can optimize your existing model.

Thank you so much for your time and consideration.

Best regards,
@fadhfaiz

Hi Alexander,

Thank you for putting your code on github, it's really impressive and amazing work. I was going through your code however I have some confusion, if you could please help me out with that.

My query is about creating data piplines using "tf.data". I debug the code and got the final dataset structure as given below:

<DatasetV1Adapter shapes: ((?, ?, ?, 3), (?, ?, ?, 1), (?, 2), ((?,), (?,))), types: (tf.float32, tf.float32, tf.int32, (tf.string, tf.string))>

(?, ?, ?, 3): Input Image
(?, ?, ?, 1): Saliency map
(?, 2) : Please explain
((?,), (?,)) : Please explain

Please if you can help me out by explaining the above two dimensions. I will deeply appreciate your help.