Memory usage about cloudmap HOT 12 CLOSED

The crash happens at this line:

    images = np.empty(shape=(5, 2, 
                             SatelliteData.outheight, 
                             SatelliteData.outwidth))

outheight and outwidth are the values height and width from the $HOME/.CreateCloudMap/CreateCloudMap.ini

So first question: Is there such a file for the user (root?) running the script?
Second question: What are the values?
Could you please try this in a python console:

import numpy as np
height = 1024
width = 2048
images = np.empty(shape=(5, 2, height, width)

(replace the values of height and width with the ones from your ini file.)
The error message looks like there is really not enough memory (and not a problem of dependencies).

from cloudmap.

Damn, that is the only setting I didn't try because the resolution is kinda fixed to 5400x2700, to go along with the xfce-planet script and cloudmap failed even before connecting and downloading any images so I just assumed it couldn't really have been a memory problem :)

But, you nailed it. I added 2G more swap space, no more memory error, ran into some issues with PIL(low), had to

ln -s /usr/lib/x86_64-linux-gnu/libz.so /usr/lib
ln -s /usr/lib/x86_64-linux-gnu/libfreetype.so /usr/lib
ln -s /usr/lib/x86_64-linux-gnu/libjpeg.so /usr/lib

pip uninstall PIL
pip install PIL --upgrade

and voila : https://apollo.open-resource.org/clouds.jpg

Thanks again :)

from cloudmap.

Commit c35f58f should reduce the required amount of memory by about a factor of 1/5.

from cloudmap.

Sweet, I'll check and report:

prior (from pip):

'''
/usr/bin/time -v create_map -c CreateCloudMap.ini
('Download image date/time: ', '2014-06-27 06:00 UTC')
Satellite file: /home/chrono/.xplanet/world/clouds/raw/2014_6_27_600_MTSAT2_4_S1.jpeg
Satellite file: /home/chrono/.xplanet/world/clouds/raw/2014_6_27_600_MET7_2_S1.jpeg
Satellite file: /home/chrono/.xplanet/world/clouds/raw/2014_6_27_600_MSG3_9_S1.jpeg
Satellite file: /home/chrono/.xplanet/world/clouds/raw/2014_6_27_600_GOES13_4_S1.jpeg
Satellite file: /home/chrono/.xplanet/world/clouds/raw/2014_6_27_600_GOES15_4_S1.jpeg
finished
Command being timed: "create_map -c CreateCloudMap.ini"
User time (seconds): 108.82
System time (seconds): 5.03
Percent of CPU this job got: 88%
Elapsed (wall clock) time (h:mm:ss or m:ss): 2:08.69
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 9053840
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 81
Minor (reclaiming a frame) page faults: 1826452
Voluntary context switches: 2843
Involuntary context switches: 585
Swaps: 0
File system inputs: 33216
File system outputs: 12624
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
'''

Running fresh git clone:

'''
/usr/bin/time -v python create_map.py -c CreateCloudMap.ini
Download image date/time: 2014-06-27 06:00 UTC
Satellite file: ./2014_6_27_600_MTSAT2_4_S1.jpeg
Satellite file: ./2014_6_27_600_MET7_2_S1.jpeg
Satellite file: ./2014_6_27_600_MSG3_9_S1.jpeg
Satellite file: ./2014_6_27_600_GOES13_4_S1.jpeg
Satellite file: ./2014_6_27_600_GOES15_4_S1.jpeg
finished
Command being timed: "python create_map.py -c CreateCloudMap.ini"
User time (seconds): 108.27
System time (seconds): 5.09
Percent of CPU this job got: 91%
Elapsed (wall clock) time (h:mm:ss or m:ss): 2:03.54
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 3582352
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 1821929
Voluntary context switches: 2477
Involuntary context switches: 274
Swaps: 0
File system inputs: 0
File system outputs: 12624
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
'''

from cloudmap.

Hi,
I have same issue on Windows and I am pushing to 8192x4096 and get
Traceback (most recent call last):
File "D:\temp\cloudimagesmap\create_map-script.py", line 9, in
load_entry_point('CreateCloudMap==0.4.2', 'console_scripts', 'create_map')()

File "C:\Python27\lib\site-packages\cloudmap\create_map.py", line 278, in main

SatelliteData.outwidth))

ValueError: array is too big.
Now I am not short of RAM or grunt, but I am on windows so getting stuff out of processes isn't great. I get the same in 0.5

Thanks

from cloudmap.

Ok. I can reproduce the problem using windows.

from cloudmap.

A work around to the problem seems to be to use http://www.cygwin.com/
But this might be just by chance (if you use the 32bit version).

If you create such large images, then a lot of memory is needed (about 2 GB which is just at the limit for 32bit processes). Now the better solution is to use 64bit Python. I was able to install it on Windows and using the 64bit version of numpy, scipy, matplotlib, pillow, and pip from here:
http://www.lfd.uci.edu/~gohlke/pythonlibs/

Then I followed the instructions from http://blog.victorjabur.com/tag/python-express-2010-visual-build-compile-windows-executable/ to setup an environment so I could compile python modules using 64bit Microsoft Visual Studio (Express) 2008 Compiler. Then simple issuing

pip install CreateCloudMap

did the trick.

Here is the list of all installed python libraries, just in case I forgot something important above:

>pip freeze
CreateCloudMap==0.5.0
DateTime==4.0.1
MarkupSafe==0.23
Pillow==2.4.0
Pygments==1.6
configobj==5.0.5
configparser==3.3.0r2
ipython==2.1.0
matplotlib==1.3.1
numpy==1.8.1
pillowfight==0.2
pycairo==1.10.0
pyproj==1.9.3
pyreadline==2.0
pyresample==1.1.0
pytz==2014.4
pyzmq==14.3.1
requests==2.3.0
scipy==0.14.0
six==1.7.2
zope.interface==4.1.1

Of course, your Windows must be a 64bit version. (On stackoverflow you can find a link to download Visual Studio 2008 Express which is not linked directly on the MS website anymore. Newer version seem not to work.)

In my case the 32bit version is crashing within the pyresample library so I do not really know a way to solve this problem in a cleverer way (e.g. splitting up the images and do the resampling sequentially).

from cloudmap.

Joachim,

I started with the 64Bit Python and was have issues with getting the libraries installed, so actually moved to 32bit. I will follow what you suggest and I will come back to you.

Ta.

P please consider the environment before printing this e-mail.
Did you know that IT are now fully ISO 3103 Compliant

From: Joachim [mailto:[email protected]]
Sent: 27 June 2014 14:42
To: jmozmoz/cloudmap
Cc: Michael Dear
Subject: [SPAM] [HEADER CHECKING] - Re: [cloudmap] Memory usage (#2) - Email found in subject

A work around to the problem seems to be to use http://www.cygwin.com/
But this might be just by chance (if you use the 32bit version).

If you create such large images, then a lot of memory is needed (about 2 GB which is just at the limit for 32bit processes). Now the better solution is to use 64bit Python. I was able to install it on Windows and using the 64bit version of numpy, scipy, matplotlib, pillow, and pip from here:
http://www.lfd.uci.edu/~gohlke/pythonlibs/http://www.lfd.uci.edu/%7Egohlke/pythonlibs/

Then I followed the instructions from http://blog.victorjabur.com/tag/python-express-2010-visual-build-compile-windows-executable/ to setup an environment so I could compile python modules using 64bit Microsoft Visual Studio (Express) 2008 Compiler. Then simple issuing

pip install CreateCloudMap

did the trick.

Here is the list of all installed python libraries, just in case I forgot something important above:

pip freeze

CreateCloudMap==0.5.0

DateTime==4.0.1

MarkupSafe==0.23

Pillow==2.4.0

Pygments==1.6

configobj==5.0.5

configparser==3.3.0r2

ipython==2.1.0

matplotlib==1.3.1

numpy==1.8.1

pillowfight==0.2

pycairo==1.10.0

pyproj==1.9.3

pyreadline==2.0

pyresample==1.1.0

pytz==2014.4

pyzmq==14.3.1

requests==2.3.0

scipy==0.14.0

six==1.7.2

zope.interface==4.1.1

Of course, your Windows must be a 64bit version. (On stackoverflow you can find a link to download Visual Studio 2008 Express which is not linked directly on the MS website anymore. Newer version seem not to work.)

In my case the 32bit version is crashing within the pyresample library so I do not really know a way to solve this problem in a cleverer way (e.g. splitting up the images and do the resampling sequentially).

—

from cloudmap.

Hi,

Yes switching to 64bit seems to work, I even pushed to 16384x8192, but it takes a long time to render. I have noticed that the timing if the script is great. eg I have asked it to download an update of the cloud maps, which I can see are @ Dundee, but the script keeps using 900 image (both 1200 and 1500 images are available). If I replace the images it will use them but it wont download them. I am guessing it is a timing issue. Below is output when run at 1634,

Create Cloud images
2048x1024
Download image date/time: 2014-07-02 09:00 UTC
Satellite file: c:\temp\cloudimagesmap\2014_7_2_900_MTSAT2_4_S1.jpeg
Satellite file: c:\temp\cloudimagesmap\2014_7_2_900_MET7_2_S1.jpeg
Satellite file: c:\temp\cloudimagesmap\2014_7_2_900_MSG3_9_S1.jpeg
Satellite file: c:\temp\cloudimagesmap\2014_7_2_900_GOES13_4_S1.jpeg
Satellite file: c:\temp\cloudimagesmap\2014_7_2_900_GOES15_4_S1.jpeg
4096x2048
Download image date/time: 2014-07-02 09:00 UTC
Satellite file: c:\temp\cloudimagesmap\2014_7_2_900_MTSAT2_4_S1.jpeg
Satellite file: c:\temp\cloudimagesmap\2014_7_2_900_MET7_2_S1.jpeg
Satellite file: c:\temp\cloudimagesmap\2014_7_2_900_MSG3_9_S1.jpeg
Satellite file: c:\temp\cloudimagesmap\2014_7_2_900_GOES13_4_S1.jpeg
Satellite file: c:\temp\cloudimagesmap\2014_7_2_900_GOES15_4_S1.jpeg
8192x4096
Download image date/time: 2014-07-02 09:00 UTC
Satellite file: c:\temp\cloudimagesmap\2014_7_2_900_MTSAT2_4_S1.jpeg
Satellite file: c:\temp\cloudimagesmap\2014_7_2_900_MET7_2_S1.jpeg
Satellite file: c:\temp\cloudimagesmap\2014_7_2_900_MSG3_9_S1.jpeg
Satellite file: c:\temp\cloudimagesmap\2014_7_2_900_GOES13_4_S1.jpeg
Satellite file: c:\temp\cloudimagesmap\2014_7_2_900_GOES15_4_S1.jpeg
16384x8192
Download image date/time: 2014-07-02 09:00 UTC
Satellite file: c:\temp\cloudimagesmap\2014_7_2_900_MTSAT2_4_S1.jpeg
Satellite file: c:\temp\cloudimagesmap\2014_7_2_900_MET7_2_S1.jpeg
Satellite file: c:\temp\cloudimagesmap\2014_7_2_900_MSG3_9_S1.jpeg
Satellite file: c:\temp\cloudimagesmap\2014_7_2_900_GOES13_4_S1.jpeg
Satellite file: c:\temp\cloudimagesmap\2014_7_2_900_GOES15_4_S1.jpeg
Press any key to continue . . .

from cloudmap.

I just tested it and this is the output:

create_map
Download image date/time:  2014-07-02 15:00 UTC
Satellite file: c:\temp\xplanet\images\2014_7_2_1500_MTSAT2_4_S1.jpeg
Satellite file: c:\temp\xplanet\images\2014_7_2_1500_MET7_2_S1.jpeg
Satellite file: c:\temp\xplanet\images\2014_7_2_1500_MSG3_9_S1.jpeg
Satellite file: c:\temp\xplanet\images\2014_7_2_1500_GOES13_4_S1.jpeg
Satellite file: c:\temp\xplanet\images\2014_7_2_1500_GOES15_4_S1.jpeg
finished

So I close this issue, because the memory problem is explained (not necessarily solved). If there is really a timing issue, then please open a new issue.

from cloudmap.

Hmm, I had the same observation today, 1200 and 1500 were very late somehow, when you run the script, it just doesn't download anything.

from cloudmap.

I have opened a new issue

from cloudmap.