Cloud Download about cloudmap HOT 11 CLOSED

I can second that since I saw the very same behavior yesterday and also a couple of days ago. Python 2.7, Ubuntu 12.04 (x86_64).

from cloudmap.

I have spent the day running the job every 30mins and only at 1459 did the script download the 1200 images.

from cloudmap.

To me it seems like 1 hour 15 past official release time (as in the filenames) seems like the best spot to set a scheduler (5 or 10 minutes past often didn't work), so I've set the cloudmap repo generator script (https://github.com/apollo-ng/cloudmap) cron to 4:15, 7:15, 10:15 [...].
I also wonder why there is a 6 hour window at night (i.e. no 0000 image) :)

from cloudmap.

You also have to factor in the images wont appear at 0900 for the 0900 image. It seems to take around 60-90mins for the images to become available. But even when they do the script doesn't seem to pick it up. Which is why I am pointing the finger at a time issue. Which from experience can be a right pain.

from cloudmap.

Ack. That confirms my observation, 75 minutes past is working almost always (see the commit log of the repo). If it's something in the code, we should be able to reproduce it. But mostly it just runs for days without any hitch.

from cloudmap.

The scripts looks for the latest time for which all images are available. The script is only downloading new satellite images, if they are not already saved on your disk. So - I think - it should not be a problem to run the script every hour (or so), because there is only very little traffic on the Dundee server (check if the file exist at https://github.com/jmozmoz/cloudmap/blob/master/cloudmap/create_map.py#L110).

I am working on an additional tool to only check for the latest images and log it. Then it will be possible to check for the availability of different images and resolutions.

from cloudmap.

Joachim,

My only other comment is that is doesn't get the 0000 image. I have left it running now for 96 hours, and it does seam consistently to only get the image 2 hours after the image time. Now I wasn't checking the actual time it was available, so it may just be a way that Dundee is doing things. But you definitely don't get the midnight image downloaded.

from cloudmap.

Your are right, there is a bug with the 00:00UTC image (file names). I will fix this as soon as possible.

from cloudmap.

0.5.1 is deployed via pip and should hopefully throw a 0000 image tonight into https://github.com/apollo-ng/cloudmap/commits/master. Let's see and close this afterwards :)

from cloudmap.

Yes :) https://github.com/apollo-ng/cloudmap/commits/master - I can confirm getting 0000.

from cloudmap.

Thank you for confirming the fix for the midnight problem. Regarding the time lag I guess this is just the processing time at Dundee.

from cloudmap.