jmozmoz / cloudmap Goto Github PK
View Code? Open in Web Editor NEWPython script to download a cloudmap from https://clouds.matteason.co.uk/ if a new one is abailable
License: GNU General Public License v3.0
cloudmap's People
Contributors
Stargazers
Watchers
cloudmap's Issues
Cannot download (all) satellite images!
Anyone else having trouble getting latest images? For the past 24h cloudmap hasn't downloaded any new image. Manual login at dundee shows latest images of today. Am I the only one with this problem?
Cannot download (all) satellite images! (again)
Hello, somewhere around Dec 4. cloudmap ceased to work, failing with error:
Cannot download (all) satellite images!
How can I make it work again?
Cloud Download
I have noticed that the update to the maps from Dundee at least, isn't consistent.
The first time it is run on a day (8:30) it downloads the 0600 images. However it is hit and miss as to whether any further images are downloaded for the day. I have got a 0900 a couple of times, but I have to manually download 1200 and later images.
Once downloaded the images are used correctly.
For example, Running the script at 1300 and 1600 yesterday only used 0900 images were downloaded, although both the 1200 and 1500 images were available. I know nothing about python, but could the issue be in either timing or the crafting of the URL? I can't find a way to get those to be printed.
I am running 64Bit 2.7 Python on a Win8.1 box. You can see in the screen grab, the script is sticking to 0600 but the 0900 is available
satellite update?
Hi there,
there is some weird output map from GEOS_15_4_S1, it looks as if the projection is a mollweide one and that seems to screw the final cloud_map.
I've done a check on the Dundee website, there are new satellite there from japan and india, maybe we could be used as a replacement?
Thanks,
Cheers.
CreateCloudMap cannot download (all) satellite images
For several weeks now, CreateCloudMap (v. 0.9.13) has not been able to download a new cloud map. It gives the following output:
create_map_logger: INFO: Search backward for images from: 2019-09-24 12:38:23.631613
Cannot download (all) satellite images!
Memory usage
Just let me thank you for developing and releasing this absolute gem of a tool first :) I was unhappy with the often failing distribution of global cloudmaps for years because I need a reliable source to feed https://github.com/chron0/xfce-planet.
After successfully testing and using cloudmap with gentoo on my local desktop for a couple of days I wanted to deploy it on our webserver to produce high resolution cloudmaps every 3 hours as an autonomous service, so that the xplanet community and whoever else can just download and use them as a free, alternative source without having the need to setup and run cloudmap themselves.
The box is basically a slice of sponsored kvm running Ubuntu 12.04.4 LTS (x86_64), but
no matter what I try, I always get:
Traceback (most recent call last):
File "create_map.py", line 347, in <module>
main()
File "create_map.py", line 278, in main
SatelliteData.outwidth))
MemoryError
trace
stat("/etc/apt/apt.conf", 0x7fff3b40ece0) = -1 ENOENT (No such file or directory)
stat("/var/lib/dpkg/status", {st_mode=S_IFREG|0644, st_size=687788, ...}) = 0
stat("/usr/bin/dpkg", {st_mode=S_IFREG|0755, st_size=253424, ...}) = 0
stat("/etc/debian_version", {st_mode=S_IFREG|0644, st_size=11, ...}) = 0
stat("", 0x7fff3b40f030) = -1 ENOENT (No such file or directory)
close(7) = 0
stat("/usr/lib/python2.7/dist-packages/apport/packaging", 0x7fff3b40f400) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.7/dist-packages/apport/packaging.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.7/dist-packages/apport/packagingmodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.7/dist-packages/apport/packaging.py", O_RDONLY) = 7
fstat(7, {st_mode=S_IFREG|0644, st_size=9179, ...}) = 0
open("/usr/lib/python2.7/dist-packages/apport/packaging.pyc", O_RDONLY) = 8
fstat(8, {st_mode=S_IFREG|0644, st_size=11186, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f452dbac000
read(8, "\3\363\r\n\2105\204Oc\0\0\0\0\0\0\0\0\3\0\0\0@\0\0\0s)\0\0\0d\0"..., 4096) = 4096
fstat(8, {st_mode=S_IFREG|0644, st_size=11186, ...}) = 0
read(8, "Return the package a file belong"..., 4096) = 4096
read(8, "kages/apport/packaging.pyt\22\0\0\0ge"..., 4096) = 2994
read(8, "", 4096) = 0
close(8) = 0
munmap(0x7f452dbac000, 4096) = 0
close(7) = 0
close(6) = 0
close(5) = 0
close(4) = 0
stat("/usr/lib/python2.7/dist-packages/apport/gettext", 0x7fff3b419630) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.7/dist-packages/apport/gettext.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.7/dist-packages/apport/gettextmodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.7/dist-packages/apport/gettext.py", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.7/dist-packages/apport/gettext.pyc", O_RDONLY) = -1 ENOENT (No such file or directory)
close(3) = 0
getcwd("/root/tmp", 1026) = 10
lstat("/root", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
lstat("/root/tmp", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/root/tmp/create_map.py", {st_mode=S_IFREG|0755, st_size=13373, ...}) = 0
access("/root/tmp/create_map.py", X_OK) = 0
stat("/root/tmp/create_map.py", {st_mode=S_IFREG|0755, st_size=13373, ...}) = 0
write(2, "Traceback (most recent call last"..., 35Traceback (most recent call last):
) = 35
write(2, " File \"create_map.py\", line 347"..., 46 File "create_map.py", line 347, in <module>
) = 46
open("create_map.py", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0755, st_size=13373, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f452dbac000
read(3, "#!/usr/bin/env python\r\n\r\nfrom PI"..., 4096) = 4096
read(3, " look = True\r\n i = "..., 4096) = 4096
read(3, "igParser.SafeConfigParser(\r\n "..., 4096) = 4096
read(3, " i = 1\r\n for satellite in "..., 4096) = 1085
read(3, "", 4096) = 0
write(2, " ", 4 ) = 4
write(2, "main()", 6main()) = 6
write(2, "\n", 1
) = 1
close(3) = 0
munmap(0x7f452dbac000, 4096) = 0
write(2, " File \"create_map.py\", line 278"..., 42 File "create_map.py", line 278, in main
) = 42
open("create_map.py", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0755, st_size=13373, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f452dbac000
read(3, "#!/usr/bin/env python\r\n\r\nfrom PI"..., 4096) = 4096
read(3, " look = True\r\n i = "..., 4096) = 4096
read(3, "igParser.SafeConfigParser(\r\n "..., 4096) = 4096
write(2, " ", 4 ) = 4
write(2, "SatelliteData.outwidth))\n", 25SatelliteData.outwidth))
) = 25
close(3) = 0
munmap(0x7f452dbac000, 4096) = 0
write(2, "MemoryError", 11MemoryError) = 11
write(2, "\n", 1
) = 1
rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7f452d780cb0}, {0x512110, [], SA_RESTORER, 0x7f452d780cb0}, 8) = 0
exit_group(1)
meminfo
MemTotal: 2051316 kB
MemFree: 257892 kB
Buffers: 265460 kB
Cached: 629452 kB
SwapCached: 0 kB
Active: 1155832 kB
Inactive: 460668 kB
Active(anon): 776140 kB
Inactive(anon): 11708 kB
Active(file): 379692 kB
Inactive(file): 448960 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 1500 kB
Writeback: 0 kB
AnonPages: 721676 kB
Mapped: 60148 kB
Shmem: 66260 kB
Slab: 121908 kB
SReclaimable: 103400 kB
SUnreclaim: 18508 kB
KernelStack: 1736 kB
PageTables: 20520 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 1025656 kB
Committed_AS: 1489560 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 12312 kB
VmallocChunk: 34359723644 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 46836 kB
DirectMap2M: 2050048 kB
DirectMap1G: 0 kB
I've tried upgrading all pip dependencies, removing ubuntu python packages that might collide, I must have rebuilt the whole pip tree 5 times by now but I see no light. Do you have any ideas what might cause this fail? Since cloudmap works without a glitch on my gentoo box I suppose it must be some dependency (and not a bug in cloudmap) but at this point I am running out of things I could try and I would really like to see this fly.
"Cannot download (all) satellite images!"
Hi there,
Been having some problems with create_map lately. I'm not sure when the issue arose, but it started at least a few weeks ago. Opening a terminal and running create_map -d gives
`create_map -d
Search backward for images from: 2017-09-05 15:15:32.190895
cannot download image: http://www.sat.dundee.ac.uk/xrit/057.0E/MET/2017/9/5/1500/2017_9_5_1500_MET7_2_S2_grid.jpeg
cannot download image: http://www.sat.dundee.ac.uk/xrit/057.0E/MET/2017/9/5/1200/2017_9_5_1200_MET7_2_S2_grid.jpeg
cannot download image: http://www.sat.dundee.ac.uk/xrit/057.0E/MET/2017/9/5/1200/2017_9_5_1200_MET7_2_S2_grid.jpeg
...`
I notice the satellite at 57E was replaced with one at 41.5E, but it seems the script it still looking for the one at 57E. I have updated to version 0.9.2.
create_map -V 0.9.2
Thanks for any help.
Future access to the Satellite Receiving Station
Hello,
found on this page:
The funding provided by NERC for the Satellite Receiving Station will end after March 2019.
For access to our website, images, data and archive after this date please contact us.
If we do not hear from you then the receiving station may close permanently.
Please drop them a line!
Either via their comment system or directly via email: [email protected].
I wrote this:
Hello,
thank you for providing this service!
I am only a hobbyist who recently started using your cloud maps.
However, there's probably a few more out there who would like to continue using this service.
Here's a script that generates a global cloudmap from your data:
github.com/jmozmoz/cloudmap
And here's a repository that offers that global map to anyone who might need it:
github.com/apollo-ng/cloudmap
The latter wouldn't even show up as traffic on your site, which is unfortunate.
The generated map is particularly useful for Xplanet, a powerful opensource application, developed since the late nineties and available for all major operating systems.
PS: and thanks to jmozmoz for providing this script!
Black stripes in polar areas
Hello,
thank you for providing this!
I have used the script from my own machine a few times, and then from here. Every time there's been some irregularities esp. in the Arctic, see e.g. here - I'm talking about the sharp-edged black areas in the upper left, and also smaller in the upper middle, slightly to the right.
Is it possible to improve this somehow?
I understand there's some limitations to how the original images can be combined into the final image, esp. around the poles. It would be great if python offered some intelligent filling of these obvious gaps (I'm just guessing here)?
Tracking bug for 0.6.0 release
New features in 0.6.0:
- Multiprocessing (of course 5 processes is best, because there 5 satellite images to re-project)
- Support for second projecting library: cartopy (until now only pyresample is supported)
- Python 3 support: At the moment cartopy supports Python 3, pyresample will add Python 3 support with the release 1.1.1 (mraspaud/pyresample@fde4205)
- Reorganization of code structure
building against system libs and external pythons
Hi there,
I have built cloudmap out of pypi and various issues showed up. Here some summary and patches proposal:
- missing deps to various python packages which are actually built-in modules. Datetime is already built-in in python2 and python3, configparser only in python3:
--- cloudmap-0.9.2/setup.py.orig 2017-10-27 17:18:52.515799486 +0200
+++ cloudmap-0.9.2/setup.py 2017-10-27 17:20:57.620080091 +0200
@@ -70,12 +70,11 @@
cmdclass['install'] = Install
if sys.version_info >= (3, 2):
- install_requires = ['pyresample', 'numpy', 'scipy', 'requests', 'datetime',
- 'pillow>=3.0.0', 'setuptools>=0.7.2',
- 'configparser>=3.5.0b1']
+ install_requires = ['pyresample', 'numpy', 'scipy', 'requests',
+ 'pillow>=3.0.0', 'setuptools>=0.7.2']
else:
- install_requires = ['pyresample', 'numpy', 'scipy', 'requests', 'datetime',
- 'pillow>=3.0.0', 'setuptools>=0.7.2', 'configparser']
+ install_requires = ['pyresample', 'numpy', 'scipy', 'requests',
+ 'pillow>=3.0.0', 'setuptools>=0.7.2', 'configparser>=3.5.0b1']
setup(
name='CreateCloudMap',
- eggs with incorrect version numbers
--- cloudmap-0.9.2/cloudmap/_version.py.orig 2017-10-27 16:50:06.144781102 +0200
+++ cloudmap-0.9.2/cloudmap/_version.py 2017-10-27 16:50:34.823468553 +0200
@@ -9,7 +9,7 @@
# versioneer-0.11 (https://github.com/warner/python-versioneer)
# these strings will be replaced by git during git-archive
-git_refnames = " (HEAD -> master, tag: 0.9.2, tag: 0.9.1)"
+git_refnames = " (HEAD -> master, tag: 0.9.2)"
git_full = "ef3df6ebcf543854c66dec14acd77a9c966b7f23"
# these strings are filled in when 'setup.py versioneer' creates _version.py
- the "pc" function in satellites.py seems to miss some informations for the proj_dict. The following fix works, but that's just a wild guess from what works in geo_dundee!
--- cloudmap-0.9.2/cloudmap/satellites.py.orig 2017-10-27 17:49:33.401520400 +0200
+++ cloudmap-0.9.2/cloudmap/satellites.py 2017-10-27 17:50:27.639033323 +0200
@@ -126,7 +126,8 @@
def pc(x_size, y_size):
from pyresample import geometry
"""Create area defintion for world map in Plate Carree projection"""
- proj_dict = {'proj': 'eqc'}
+ proj_dict = {'a': '6378137.0', 'b': '6356752.3',
+ 'h': '35785831.0', 'proj': 'eqc'}
area_extent = (-20037508.34, -10018754.17, 20037508.34, 10018754.17)
pc = geometry.AreaDefinition('pc', 'Plate Carree world map', 'pc',
proj_dict, x_size, y_size, area_extent)
Hope it helps!
Cheers,
Chris.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.