aliflux / maptilesdownloader Goto Github PK
View Code? Open in Web Editor NEWA super easy to use map tiles downloader built using Python
License: MIT License
A super easy to use map tiles downloader built using Python
License: MIT License
Looks like you have some Windows OS specific code...
$ python3 server.py
Starting Server...
Running Server...
Traceback (most recent call last):
File "server.py", line 167, in <module>
run()
File "server.py", line 162, in run
os.startfile('UI\\index.htm', 'open')
AttributeError: module 'os' has no attribute 'startfile'
Actually, I'm a newby working with Map using Cesium. I found that Cesium needs a map provider as WMS and I've tried MapTileDownloader and it works great except it cannot output images as Web Map Tiles Service (WMTS or WMS) which is the format that Cesium supports.
I cannot find any software that can export map tiles in WMS as well.
So, how can we add an option to the downloader the tiles for WMS?
Thanks.
Thanks for that really nice peace of work!
I use it to fill up a simple nginx-cache, that buffers tiles from OpenTopoMap.
(OpenTopoMap is slooooow, so the cache makey my life easier).
For that I modified main.js and added the caches URL as tile-server.
While downloading with MapTilesDownloader many errors occur. Most of them are "Error while relaying tile".
What exactly this error means?
Line 28 in 69f48a7
The OSMF tile usage policy states
In particular, downloading significant areas of tiles at zoom levels 17 and higher for offline or later usage is forbidden without prior consultation with a System Administrator.
I didn't identify MapTilesDownloader as a heavy user in openstreetmap/operations#371, but it's still doing something not allowed.
Yandex map uses EPSG3395 CRS.
https://core-renderer-tiles.maps.yandex.net/tiles?l=map&x={x}&y={y}&z={z}
MTD downloads tiles incorrectly.
does this look pretty accurate?
https://www.rainviewer.com/api/geotiff.html#:~:text=It%20is%20quite%20easy%20to%20convert%20a%20PNG,-co%20compress%3Dlzw%20-of%20Gtiff%20-a_srs%20EPSG%3A3857%20%7Bsource_image%7D%20%7Boutput_image%7D
I am also trying to see if geotiff can then be converted to a different format, say NAIP.
Hello,
I made a fantasy game map (not on earth) using QGIS and mapbox (for style) and I'd like to get offline tiles of my map.
How could I add mapbox provider for instance to get tiles?
Thank you
Running Windows 10, can get the app loaded and select area but no download.
Exception happened during processing of request from ('127.0.0.1', 11709)
Traceback (most recent call last):
File "C:\Users\cwilson.KORBEN\AppData\Local\Programs\Python\Python36\lib\socketserver.py", line 639, in process_request_thread
self.finish_request(request, client_address)
File "C:\Users\cwilson.KORBEN\AppData\Local\Programs\Python\Python36\lib\socketserver.py", line 361, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "C:\Users\cwilson.KORBEN\AppData\Local\Programs\Python\Python36\lib\socketserver.py", line 696, in init
self.handle()
File "C:\Users\cwilson.KORBEN\AppData\Local\Programs\Python\Python36\lib\http\server.py", line 418, in handle
self.handle_one_request()
File "C:\Users\cwilson.KORBEN\AppData\Local\Programs\Python\Python36\lib\http\server.py", line 406, in handle_one_request
method()
File "server.py", line 153, in do_POST
self.writerByType(outputType).addMetadata(lock, os.path.join("output", outputDirectory), filePath, outputFile, "Map Tiles Downloader via AliFlux", "png", boundsArray, centerArray, minZoom, maxZoom, "mercator", 256 * outputScale)
AttributeError: 'NoneType' object has no attribute 'addMetadata'
...
127.0.0.1 - - [29/Apr/2020 00:15:04] "POST /download-tile HTTP/1.1" 200 -
HIT: https://a.tile.openstreetmap.org/{z}/{x}/{y}.png
RETURN: 429
...
You need find a way for add header at request 'urllib.request.urlretrieve', like this: {"User-Agent", "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0"} ?
Your code should seem like a Webbrowser for OSM servers, not a script.
Hi!
I tried to use your script for downloading map tiles. I got an error after selecting a region and trying to download the tiles.
"9652,12319,15 : Error while relaying tile
All requests are done"
KeyError: 'CONTENT-LENGTH'
Any idea of what the issue is here? I have no experience in coding etc. Would be great to hear back from you!
Krzysztof
An error occurred when I opened the βhttp://localhost:8080/β
FileNotFoundError: [Errno 2] No such file or directory: './UI/favicon.ico'.
I searched the entire engineering project but couldn't find it.
I pray you can give me a solution.
Hi Great program here.... thanks
When i edit to other mapbox style in Main.js - the map stop loading and blank mapview - do i do something wrong or ??
Thanks
Frank
Love this app! However, when dealing with tens of thousands of tiles in a single go, the browser UI eventually crashes (Chrome says, "Aw, Snap! Something went wrong while displaying this webpage."). I'd say 50k tiles seems to be the limit.
I thought maybe it's a memory leak related to the ever-expanding log-view container. But I commented out the logItemRaw() function and it still happens. It's worth noting that the logging definitely slows down the app over time. It would be better to implement a fixed sized queue that only shows the last 25 log events.
When using the tool for larger tilesets, I ran into some easily fixable issues.
During downloading the process gets appended in the textarea in the sidebar. This causes lagging until the page reacts very slow or even freezes complitely after some time. The issue here is a very long text inside the textarea.
You can work around it, if you limit the content to a certain length, which can be achived with a small call to a substring function like so:
function logItemRaw(text) {
var logger = $('#log-view');
var value = logger.val() + '\n' + text;
var maxLength = 10000;
if(value.length > maxLength){
value = value.substring(value.length - maxLength, value.length);
}
logger.val(value);
logger.scrollTop(logger[0].scrollHeight);
}
When downloading for longer periods of time, I ran into an issue, where the tab in Chrome crashes, with an 'Out of memory' message. I noticed that the time until this happens it is related to how large the image tiles are.
The problem is that every request with its result (which also contains the base64 tile) is stored in an array but never removed again. This causes a constant memory increase over time.
You can fix this by removing the data from the array, when it is not necessary anymore.
This can be achived by these lines inside the 'always' function of the ajax request:
var removeIndex = requests.indexOf(request);
if(removeIndex >= 0) requests.splice(removeIndex, 1);
The complete part sould look like this:
.always(function(data) {
i++;
removeLayer(boxLayer);
updateProgress(i, allTiles.length);
done();
var removeIndex = requests.indexOf(request);
if(removeIndex >= 0) requests.splice(removeIndex, 1);
if(cancellationToken) {
return;
}
});
All in all I have to say that your tool is great! Not only by getting the job done, but doing it in a beautifully visualized way!
I could watch it download all day long :)
Thanks for publishing it π
As mentioned in my last issue I just want to fill my cache and don't need the downloaded tiles.
Do you see an easy way for me to NOT create the output on filesystem?
Hi there, I want to archive all the tiles on an encient tile provider (since I am afraid that they will shut down their tile server and I have an app that I cannot change that is still using them)
the tile server is here: http://tiles.cdnst.net/base/
maps are organized in /z/x/y.png (I think)
and are using modestmaps-as3 (Modest Maps)
examples: http://tiles.cdnst.net/base/6/34/23.png
http://tiles.cdnst.net/base/6/34/24.png
http://tiles.cdnst.net/base/6/34/25.png
http://tiles.cdnst.net/base/6/34/26.png
http://tiles.cdnst.net/base/6/34/27.png
http://tiles.cdnst.net/base/6/34/28.png
http://tiles.cdnst.net/base/6/34/29.png
http://tiles.cdnst.net/base/6/34/30.png
http://tiles.cdnst.net/base/6/35/24.png
http://tiles.cdnst.net/base/7/35/24.png
so does someone know how can I download them all (including all zoom settings)?
if downloading is not an option, can I just render them and make them look like the above?
Thanks for Anwsering and Best Regards
I'd like to be able to combine the output into one high res image, or better yet, have that as an output option. Are you willing to implement this, or have any recommendations on how I would go about doing this?
Traceback (most recent call last):
File "C:\Users\aaves\Desktop\maptiles\src\server.py", line 236, in
run()
File "C:\Users\aaves\Desktop\maptiles\src\server.py", line 228, in run
httpd = serverThreadedHandler(server_address, serverHandler)
File "C:\Users\aaves\AppData\Local\Programs\Python\Python39\lib\socketserver.py", line 452, in init
self.server_bind()
File "C:\Users\aaves\AppData\Local\Programs\Python\Python39\lib\http\server.py", line 138, in server_bind
socketserver.TCPServer.server_bind(self)
File "C:\Users\aaves\AppData\Local\Programs\Python\Python39\lib\socketserver.py", line 466, in server_bind
self.socket.bind(self.server_address)
OSError: [WinError 10013] An attempt was made to access a socket in a way forbidden by its access permissions
@AliFlux I think this ability is not working, because if I start downloading, stop, and after that start again it is downloading already downloaded files
nice work btw π
Whenever the tool tries to download the tiles I'm getting error "error while relaying tile". In the python console the error output looks like so:
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 59134)
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\lib\socketserver.py", line 650, in process_request_thread
self.finish_request(request, client_address)
File "C:\ProgramData\Anaconda3\lib\socketserver.py", line 360, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "C:\ProgramData\Anaconda3\lib\socketserver.py", line 720, in __init__
self.handle()
File "C:\ProgramData\Anaconda3\lib\http\server.py", line 426, in handle
self.handle_one_request()
File "C:\ProgramData\Anaconda3\lib\http\server.py", line 414, in handle_one_request
method()
File "server.py", line 81, in do_POST
postvars = parse_multipart(self.rfile, pdict)
File "C:\ProgramData\Anaconda3\lib\cgi.py", line 220, in parse_multipart
headers['Content-Length'] = pdict['CONTENT-LENGTH']
KeyError: 'CONTENT-LENGTH'
----------------------------------------
How can i get Jpeg output
Hi,
How we can add apple map service to this project? I want to download apple map tiles for a location.
How to change the map source and add your own map source published with mapbox
When loading the app, the map screen allowing you to select a geographical area no longer works. The following is displayed in the CMD prompt.
Exception occurred during processing of request from ('127.0.0.1', 50470)
Traceback (most recent call last):
File "C:\Users\alanr\AppData\Local\Programs\Python\Python311\Lib\socketserver.py", line 691, in process_request_thread
self.finish_request(request, client_address)
File "C:\Users\alanr\AppData\Local\Programs\Python\Python311\Lib\socketserver.py", line 361, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "C:\Users\alanr\AppData\Local\Programs\Python\Python311\Lib\socketserver.py", line 755, in init
self.handle()
File "C:\Users\alanr\AppData\Local\Programs\Python\Python311\Lib\http\server.py", line 432, in handle
self.handle_one_request()
File "C:\Users\alanr\AppData\Local\Programs\Python\Python311\Lib\http\server.py", line 420, in handle_one_request
method()
File "C:\Users\alanr\Desktop\MapTilesDownloade\src\server.py", line 219, in do_GET
with open(file, "rb") as f:
^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: './UI/favicon.ico'
In the method "gettilerect"οΌmain.jsοΌ, C1 represents northwest and C2 represents southeast. C1 and C2 are stored in mapboxgl.lnglatbounds, but the two parameters in lnglatbounds should be southwest and northeast. Is it wrong to use them? Still didn't explain clearly
I've cloned the repo and tried to run:
python3 ./src/server.py
but got a
ModuleNotFoundError: No module named 'PIL'
error.
Do I need to install packages like in the javascript world?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.