I've just found this project, and took a look at some of those online examples. The first thing I noticed is that mouse wheel actions are reversed. In other words, scrolling up does a zoom out, while in Google Maps and most other mapping software, it zooms in.

Similar to how I cache chunks when rendered and only re-render them if they change, I should do the same for tiles. This will greatly speed up generating all the tiles subsequent times (although not the first time)

It'd be great if http://github.com/brownan/Minecraft-Overviewer or the README had a link to a live working demo map so that folks can quickly see what the program's output looks like.

A couple of days ago I was able to get a map rendered (http://novylen.net/~minecraft/) (I've now removed the outlying chunks at ~1000000,0 btw, so it needs to recalculate the zoom levels). On attempting to do an incremental update, it got through rendering the new/updated chunks, and then complained that the map was now too large. :(

I suppose I crossed some magic threshold somewhere.

178000/178830 chunks rendered
Done!
Traceback (most recent call last):
File "gmap.py", line 70, in
main()
File "gmap.py", line 46, in main
q = quadtree.QuadtreeGen(w, destdir)
File "/home/minecraft/brownan-Minecraft-Overviewer-72090ff/quadtree.py", line 51, in init
raise ValueError("Your map is waaaay to big!")
ValueError: Your map is waaaay to big!

Add lighting

Entering on behalf of a few user comments on the SA MC thread. In short, they expect the community-accepted north (the direction clouds move towards) to be either upper left or upper right. Currently, north is the lower-left corner of the map.

My own suggestion would be for north being in the upper-left corner, as most people are right-handed and that's the way one would normally orient a piece of paper for reading/writing (obviously not to the extreme of the isometric view, but you get the idea.)

On Google maps, you can do annotations - put up markers with comments. Naming towns, for instance. I'd probably write a script that put all existing member-rank warp points up on the map by coordinate, if possible.

I'd like to be able to do that. Low priority compared to bugfixes but very nice.

I noticed gmap.py deletes old tiles at the start now. Since our map takes 10hrs to generate fully it means I can't generate the map on top of the live copy without 10hrs of it being mostly missing. Can you please change this so tiles are only deleted when the new copy is ready to put in place?

A cave mode image renders the bottom of the oceans, which can block cave systems. Right now the cave renderer detects caves by only rendering blocks which have no sunlight hitting them (which is easy since sunlight info is stored in the datafiles). But apparently no sunlight hits the bottom of the ocean either. Perhaps a hard-coded exception is in order.

i'm not really sure how to even describe this problem, but have a look:

http://smp.em32.net/map.bug/

here's a screenshot in case what i'm seeing isn't what you're seeing: http://smp.em32.net/map.bug/ss.png

i didn't change (as far as i can tell) how i ran overviewer. the only thing that i can think of that might have changed between the execution that generated this buggy map and the prior one is that my world may have been identical (that is, no updated chunks).

Now that I'm using google maps, I'd like to put in mineral overlays for various minerals, so one can tell where they are.

to facilitate the rendering of really large maps, it would be nice to support parallelisation on multiple nodes in a cluster (assuming some type of shared storage).

chunk rendering seems easy to do. not so sure about image tiling.

I should check to see if a tile is blank and just... not save it. Right now it generates probably hundreds of completely transparent tiles especially around the edges that take up space.

Add a switch to get a topdown view.

In the event the entire map needs to be re-rendered when running gmap.py as an update, any time there is a tile from the previous render and the new tile would be blank, the old tile is left in place, making for some erroneous displays.

To demonstrate: Create a world, leave it small. Render it. Go running in one direction. Re-render. The old view would fill up most of the bounds, leaving few blank tiles. The new world will have many blank tiles, since you only went in one direction. Where the new blanks overlap with the old non-blanks, you get the old tiles.

It's become apparent to me that, instead of splitting the world in half, I need to have the center point be anchored somehow, probably to the origin at chunk (0,0)

Why? If the world expands one or two units in either direction, the new midpoint will shift, and all tiles will be re-generated since the existing tiles no longer match with the new tiles.

Instead of doing an initial split at the midpoint, I think I'll change it to split at the origin.

The worker processes die but the master process does not.

Instead of searching $CWD, search the same directory as texture.py is in.

Patch:
~/Minecraft-Overviewer $ git diff
diff --git a/textures.py b/textures.py
index b857781..c9546e4 100644
--- a/textures.py
+++ b/textures.py
@@ -9,9 +9,10 @@ import numpy
from PIL import Image, ImageEnhance

 def _get_terrain_image():
-    # Check the current directory for terrain.png first:
-    if os.path.isfile("terrain.png"):
-        return Image.open("terrain.png")
+    # See if terrain.png is in the application directory
+    t = os.path.join(os.path.abspath(os.path.dirname(sys.argv[0])),"terrain.png")
+    if os.path.isfile(t):
+        return Image.open(t)

     if "darwin" in sys.platform:
         # On Macs, terrain.png could lie at

For sprites like flowers and torches, they are still rendered as a block. I need to put in an exception for these blocks so I don't get a cube with a torch drawn on it.

Going ahead and putting an issue in for this. I got most of the way through getting py2exe to work, but then my computer died.

Making an exe actually isn't that hard, but there are a couple changes to the code I need to make. Easy stuff.

1. add multiprocessing.freeze_support() to the beginning of the entry point in gmap.py
2. Eliminate references to the built in global var __file__, since it's undefined for frozen executables.

It's easy to generate a separate set of tiles, that's just one flag. But I'd like to figure out how to generate a separate layer and toggle between them.

Running py-2.7, the following command:

python C:\Users\Munki\Desktop\Minecraft-Overviewer\gmap.py -p 4 C:\Users\Munki\AppData\Roaming.minecraft\saves\World1\ C:\Users\Munki\Desktop\World1\

Produces the following traceback:

Traceback (most recent call last):
File "C:\Users\Munki\Desktop\Minecraft-Overviewer\gmap.py", line 76, in
main()
File "C:\Users\Munki\Desktop\Minecraft-Overviewer\gmap.py", line 60, in main
world.generate_quadtree(results, mincol, maxcol, minrow, maxrow, tiledir, options.procs)
File "C:\Users\Munki\Desktop\Minecraft-Overviewer\world.py", line 382, in generate_quadtree
quadtree_recurse(chunkmap, colstart, colend, rowstart, rowend, prefix, "base", sem)
File "C:\Users\Munki\Desktop\Minecraft-Overviewer\world.py", line 547, in quadtree_recurse
quad0result.start()
File "C:\Users\Munki\Desktop\Minecraft-Overviewer\world.py", line 668, in start
multiprocessing.Process.start(self)
File "C:\Python27\lib\multiprocessing\process.py", line 104, in start
self._popen = Popen(self)
File "C:\Python27\lib\multiprocessing\forking.py", line 244, in init
dump(process_obj, to_child, HIGHEST_PROTOCOL)
File "C:\Python27\lib\multiprocessing\forking.py", line 167, in dump
ForkingPickler(file, protocol).dump(obj)
File "C:\Python27\lib\pickle.py", line 224, in dump
self.save(obj)
File "C:\Python27\lib\pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "C:\Python27\lib\pickle.py", line 419, in save_reduce
save(state)
File "C:\Python27\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
save(v)
File "C:\Python27\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 562, in save_tuple
save(element)
File "C:\Python27\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
save(v)
File "C:\Python27\lib\pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "C:\Python27\lib\pickle.py", line 419, in save_reduce
save(state)
File "C:\Python27\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
save(v)
File "C:\Python27\lib\pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "C:\Python27\lib\pickle.py", line 419, in save_reduce
save(state)
File "C:\Python27\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
save(v)
File "C:\Python27\lib\pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "C:\Python27\lib\pickle.py", line 396, in save_reduce
save(cls)
File "C:\Python27\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 748, in save_global
(obj, module, name))
pickle.PicklingError: Can't pickle <type 'thread.lock'>: it's not found as thread.lock
Traceback (most recent call last):
File "", line 1, in
File "C:\Python27\lib\multiprocessing\forking.py", line 347, in main
self = load(from_parent)
File "C:\Python27\lib\pickle.py", line 1378, in load
return Unpickler(file).load()
File "C:\Python27\lib\pickle.py", line 858, in load
dispatchkey
File "C:\Python27\lib\pickle.py", line 880, in load_eof
raise EOFError
EOFError

For the tiles that are blank and thus not saved, the gmaps interface still tries to load those tiles and produces a 404. If the map were published online, this could lead to a good deal of unnecessary server requests.

Going through the code, I noted some definite opportunities for improvement:

1. The directory structure is badly balanced. Currently, each folder will have a maximum of 12 files in it, while optimal would be ~300 files per directory. Having several zoom levels in the same directory (underscores!) would fix this.

2. .hash is a bad idea. Filesystems generally do not handle thousands of 16 byte files gracefully-- a dictionary, pickled on completion or KeyboardInterrupt, would be much faster, and 41,000 entries is a small dictionary. Use multiprocessing.Monitor to handle the shared memory. (N.B.: thousands of ~5KB files make minecraft saves load much slower than they would if instead of tiny 16x16x128 chunks they were 64x64x128 chunks)

3. Chunk rendering could be improved. In particular, ChunkRenderer._find_oldimage gives horrible O(n^2) performance-- just determine the directory directly from the chunk's xPos and zPos attributes (it's base36, I have some code snippets if you'd fine them useful). Saving the rendered chunks to a separate directory (the output one?) would be a good idea as well (this may already be a feature)

If you cancel a render, a chunk image file may be half written. Next time, it will see it exists and try to open it, and the imaging library will crash. I need to catch this and re-render the chunk.

I currently run a SMP server with a 250MB save folder, after a few hours of rendering, it produced a google map that worked, but had something like 40,000 files and took up 1.5GB of disk space. When I cut down the zoom levels, it just cropped the map, rather than just not allowing you to zoom in as far.

This had the desired effect of reducing disk usage of the map, but didn't actually show the whole map. It would be nice to have an option for this to account for really large servers.

With the recent re-structuring of the program from recursive to iterative, the memory requirements went up a bit. I don't know how much exactly, it may have been negligible, but I wanted to open this issue to get feedback on it.

I have a couple ideas to improve memory efficiency, and I can also change it so that otherwise empty directories are not created saving disk space. But if none of this is a problem, then I'd rather focus on other things.

alexjurkiewicz, you were going to do a test on memory? Should I make improving memory usage a priority?

It looks like maybe they just need to be added to the list of transparent blocks. Cactuses aren't actually full blocks, but are instead 1 or 2 texels narrower.

The effect in the renderer is that the side faces are all the way out to fill the block, and the top face is surrounded by a narrow ring of sand, because the texture isn't expected to extend out that far on the top (but it is correct on the bottom.)

The dirt block underneath a planted sapling isn't rendered, making it appear the sapling is floating above a hole. Readily visible if you plant a group of saplings all touching each other on a layer of dirt 1 block deep, with stone or something recognizable underneath.

Has the neat effect of making it look like everything is covered in several feet of snow, but probably isn't desired. ;) I think snow is maybe actually a 1/4 block (could be checked by letting some snow fall next to a half-block.)

currently Overviewer requires write permissions to the world directory to write out the rendered chunks. this makes rendering a read-only copy of a world impossible (for example, a read-only ZFS snapshot).

RFE: add a new option --cache-dir to allow the user to specify a writable location for the chunk cache

I want to put in an exception to not render water block faces that are adjacent to another water block. This should fix the odd transparency issues.

README claims output directory must exist prior to running gmap.py; gmap.py disagrees:

if not os.path.exists(destdir):
    os.mkdir(destdir)

I'm lost, could be an issue, could be me.

5900/16384 tiles complete on level 1/7
6000/16384 tiles complete on level 1/7
Traceback (most recent call last):
File "/usr/share/minecraft-overviewer/quadtree.py", line 42, in newfunc
return func(_args, *_kwargs)
File "/usr/share/minecraft-overviewer/quadtree.py", line 515, in render_worldtile
tileimg.paste(chunkimg.convert("RGB"), (xpos, ypos), chunkimg)
File "/usr/lib/python2.6/dist-packages/PIL/Image.py", line 679, in convert
self.load()
File "/usr/lib/python2.6/dist-packages/PIL/ImageFile.py", line 215, in load
raise_ioerror(e)
File "/usr/lib/python2.6/dist-packages/PIL/ImageFile.py", line 52, in raise_ioerror
raise IOError(message + " when reading image file")
IOError: unrecognized data stream contents when reading image file
Traceback (most recent call last):
File "/usr/share/minecraft-overviewer/gmap.py", line 123, in
main()
File "/usr/share/minecraft-overviewer/gmap.py", line 82, in main
q.go(options.procs)
File "/usr/share/minecraft-overviewer/quadtree.py", line 242, in go
result.get()
File "/usr/lib/python2.6/multiprocessing/pool.py", line 422, in get
raise self._value
IOError: unrecognized data stream contents when reading image file

Since water is only rendered as a single top texture it looks very odd when it's vertical, like with a waterfall or when a body of water is at the edges of a map.

If one changes their texture pack and re-renders the tiles, the cache will prevent the tiles from being re-rendered. Even if the chunk cache is cleared, the hashes will be the same and the tiles will not change.

I should put in a function to erase the hash files from the dest directory, forcing all the tiles to be rerendered.

A minor feature request that might appeal to some:

I update the map daily at 2am, it would be nice to have some way of indicating updated chunks, perhaps through a lighter colour. Some kind of transform that gets removed the next time it processes.

Doing it by date would work, too - any tile newer than a day could be highlighted.

Traceback in here:
http://www.quartertothree.com/game-talk/showthread.php?p=2355452#post2355452

I'm not 100% sure it's a PIL issue, but that suggests it is. Could you add the ability to split the map into a number (4) of slices that can be rendered separately?

I know about this issue, just so everyone knows. Munki is working on updating it for Gmaps API v3. Hopefully that will improve things.

Some situations can leave a single process to finish a large branch of the recursive tree, while the other processes wait for it.

Under normal circumstances, the tile generation algorithm is recursive. A shared semaphore is used to determine if a recursive call can spawn a new process, or if it should be done "inline" (in the same process).

Each recursive call makes 4 calls to itself (to process each quadrant). The first 3 check the semaphore and may possibly spawn a subprocess to complete that task if a process is available. The 4th call is always done inline.

The problem is: what if recursive call 1 is really expensive, but calls 2, 3, and 4 are cheap? With 2 processes allowed, it will spawn a process for the first call, and do the other 3 itself. Since the other 3 are cheap, it will finish quickly. Then it will sit and wait while the first call does the expensive part all by itself.

I've described the issue, now to think of a solution!

From 2de06046fa19e10b4e682895c9f06cf44eacc24f Mon Sep 17 00:00:00 2001
From: Alex Jurkiewicz [email protected]
Date: Sun, 29 Aug 2010 14:59:16 +1000
Subject: [PATCH] Add support for opening terrain.png in CWD. Useful for servers running on headless boxes without the client unpacked.

textures.py | 5 +++++
1 files changed, 5 insertions(+), 0 deletions(-)

diff --git a/textures.py b/textures.py
index f2ad7cc..4224c96 100644
--- a/textures.py
+++ b/textures.py
@@ -9,6 +9,11 @@ import numpy
from PIL import Image, ImageEnhance

def _get_terrain_image():

• if os.path.isfile('terrain.png'):

•    textures = open("terrain.png","r")
  

•    buffer = StringIO(textures.read())
  

•    return Image.open(buffer)
  

  if "darwin" in sys.platform:
  # On Macs, terrain.png could lie at
  # "/Applications/minecraft/terrain.png" for custom terrain. Try this

  1.7.0.4

Wouldn't it be bangin' to use celery and a real task queue to do the heavy lifting? It'd be dead simple to spin up new worker nodes on as many different machines as one has. I think this is the real way to go for people that want to run this on a server.

I don't think I want celery to be a requirement though. It requires an actual AMQP server such as RabbitMQ to be set up, and that makes this program just that much harder to use. Maybe I'll make it an option, or make a celery branch that uses celery tasks.

Thoughts?

This will improve cachability when people add Expires headers to the tiles.

Hey, to start off, so glad you added the cachedir option. I've just about got this running but ran into a brick wall here and I don't know python too well.

Upon searching all the dirs you have listed it won't find my minecraft.jar file. Is there someway I can specify a static location. Even when putting it in the working folder or $HOME/.minecraft/bin/
I can't get it past this point.

Here's the output I get.
Traceback (most recent call last):
File "/usr/share/minecraft-overviewer/gmap.py", line 27, in
import world
File "/usr/share/minecraft-overviewer/world.py", line 23, in
import chunk
File "/usr/share/minecraft-overviewer/chunk.py", line 23, in
import textures
File "/usr/share/minecraft-overviewer/textures.py", line 105, in
terrain_images = _split_terrain(_get_terrain_image())
File "/usr/share/minecraft-overviewer/textures.py", line 88, in _get_terrain_image
return _load_image("terrain.png")
File "/usr/share/minecraft-overviewer/textures.py", line 83, in _load_image
fileobj = _find_file(filename)
File "/usr/share/minecraft-overviewer/textures.py", line 79, in _find_file
raise IOError("Could not find the file {0}. Is Minecraft installed? If so, I couldn't find the minecraft.jar file.".format(filename))
IOError: Could not find the file terrain.png. Is Minecraft installed? If so, I couldn't find the minecraft.jar file.

Thanks a million. I love what you're doing.

As follows:

$ python ~/Desktop/Minecraft-Overviewer/gmap.py -p 1 ~/Library/Application\ Support/minecraft/saves/World5/ ~/Sites/MCMap/
Scanning chunks
Rendering chunks
Rendering chunks synchronously since you requested 1 process
0/1168 chunks rendered
1/1168 chunks rendered

...(snip)...

1166/1168 chunks rendered
1167/1168 chunks rendered
Traceback (most recent call last):
File "/Users/munki/Desktop/Minecraft-Overviewer/gmap.py", line 76, in
main()
File "/Users/munki/Desktop/Minecraft-Overviewer/gmap.py", line 47, in main
results['pool'].join()
KeyError: 'pool'

When generating a set of google map tiles, the routine is currently serial, and can take quite a while. I'd like to use multiprocessing to render the tiles in parallel, this will probably speed things up quite a bit.

(For anyone wondering, the chunk rendering is parallel, but the tile rendering is not)