mide / minecraft-overviewer Goto Github PK

like mob spawn and biome details and slime

Hi there, I set the environment variable MINECRAFT_VERSION="1.15.2", but right after starting the docker container I get

mcov     | Traceback (most recent call last):
mcov     |   File "/home/minecraft/download_url.py", line 39, in <module>
mcov     |     print(get_minecraft_download_url(args.version))
mcov     |   File "/home/minecraft/download_url.py", line 23, in get_minecraft_download_url
mcov     |     raise RuntimeError("Couldn't find Minecraft Version {} in manifest file {}.".format(version, MANIFEST_URL))
mcov     | RuntimeError: Couldn't find Minecraft Version "1.15.2" in manifest file https://launchermeta.mojang.com/mc/game/version_manifest.json.
mcov exited with code 1

There is internet access, and the version exists in the manifest file... Not sure what to do here, please help!

My problem is that when the render reach 99% the output stops, but the container/process/script does not die.

The python3 processes lay dormant and do seemingly nothing (between 1 and 10% cpu utilisation) in all eternity.

I tried killing the process when it's done (when it gets stuck at 99%), but when I do that, the script leaves some type of trace that it did not end in a clean way. The next run will then pick up on this and invalidates the previous run because it "broke". Then it will start rendering the WHOLE world all over again even though only a very few couple of chunks changed since the last render.

All I want is for the script/container to exit nicely when it is done so that I can run it every now and then without having to render unchanged parts of the world again.

Otherwise a great image! Thanks for providing it :)

See https://github.com/overviewer/Overviewer-Test-Data

I recently made a couple maps with overviewer, but tried to make another and this happened. Just wanted to know if there is a fix or if I should re download overviewer.

This is a feature request to add an environment variable to only run genpoi. It should be pretty straightforward to add that this feature. I like to run gen poi more often since it's fast so I can update player locations faster. I could add this feature

If the map is accessed publicly, it needs a google maps api key. Overviewer doesn't directly have this feature, but using sed you can add the key to the generated index.html.

I notice that in the logs the container is looking for the end and nether world. Though my world name is survival. So I had my bind mount as ${HOME}/Docker/mcnew/mc-server-1/survival:/home/minecraft/server/world

Is there an elegant way to get my nether and end on the over-viewer as well?

Using the Google Maps API key is outdated now that they have switched to Leaflet. I'm not sure what else needs to be done; it looks like you already always pull the most up-to-date version of Overviewer.

2020-05-22 16:04:27 (10.0 MB/s) - '/home/minecraft/.minecraft/versions/1.15.2/1.15.2.jar' saved [15531492/15531492]

2020-05-22 16:04:27 Welcome to Minecraft Overviewer!

2020-05-22 16:04:27 E An error has occurred. This may be a bug. Please let us know!
See http://docs.overviewer.org/en/latest/index.html#help

This is the error that occurred:
Traceback (most recent call last):
File "/usr/bin/overviewer.py", line 650, in
ret = main()
File "/usr/bin/overviewer.py", line 443, in main
assetMrg = assetmanager.AssetManager(destdir, config.get('customwebassets', None))
File "/usr/lib/python3.5/dist-packages/overviewer_core/assetmanager.py", line 47, in init
self.fs_caps = get_fs_caps(self.outputdir)
File "/usr/lib/python3.5/dist-packages/overviewer_core/files.py", line 27, in get_fs_caps
return {"chmod_works": does_chmod_work(dir_to_test),
File "/usr/lib/python3.5/dist-packages/overviewer_core/files.py", line 38, in does_chmod_work
f1 = tempfile.NamedTemporaryFile(dir=dir_to_test)
File "/usr/lib/python3.5/tempfile.py", line 688, in NamedTemporaryFile
(fd, name) = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
File "/usr/lib/python3.5/tempfile.py", line 399, in _mkstemp_inner
fd = _os.open(file, flags, 0o600)
PermissionError: [Errno 13] Permission denied: '/home/minecraft/render/tmp93nlol0v'

It appears that chunks touched by the server during a render may be swapped or otherwise corrupted: overviewer/Minecraft-Overviewer#1178.

A workaround is to RCON into the server and run /save-off and /save-all before the render and run /save-on after the render.

The nether does not currently render. It's likely a bad configuration.

The CI tests are not currently running.

It wasn't super hard to find but would still be nice :)

According to discord I believe the main overviewer code has been update so overviewer doesn't error out on 1.15 worlds. It doesn't contain the new blocks but it will at least run and not crash I believe.

Can we switch to a Multi-Stage Docker build? Initial attempts left out some files which caused runtime problems. We need to also determine what are build dependencies vs runtime dependencies.

Every time I run this container, I just execute ‘docker start container-name’, it downloads the Minecraft JAR, is there a way to fix? Thanks!!

If/When #91 is merged, we'll want to make sure this image gets regularly re-built, as we want to continue to pick up the Git changes from the upstream project.

Testing automation

Hello,

I have install this overviewer version by your docker everything went fine until i run your command to render.

I'm running like this

docker run
--rm -it marctv/overviewer113
-v [world-directory]:/tmp/world/:ro
-v [config.py-directory]/config.py:/tmp/config/config.py:ro
-v [render-directory]:/tmp/export/:rw

in return i get this
"No level.dat file in '/home/minecraft/server/world'. Are you sure you have the right path?"

I search for the tmp folder and they werent there, so i create them to force, but the error keeps.
Also copy the file config.py to the tmp folder and nothing.

Can you give me a proper command example maybe i'm making something wrong
Hope you can help

Feature: Ability to have a container that periodically runs and updates the map. Currently I set this up externally but would be nice to have inside the container and keep it long lived.

Examples

Document how to get permissions correct. Discussion here: #51 (comment).

Hi, @mide! first of all, thank you for your amazing job with this repo!

I'm currently running a Minecraft server on arm64. To run the overviewer container I had to clone your repo and re-build the container directly in arm64 (RHEL8).

It worked flawlessly without any modification to the Dockerfile (the ubuntu debian base image is available in arm64) and I was able to schedule a systemd timer to run overviewer every 12 hours. The only problem is that I have to pull the changes and build the container manually every time an update is needed.

Design proposal:

1. Add an action in the .yaml workflow to use docker/setup-qemu-action@v1
2. Add An action in the .yaml workflow to use docker/setup-buildx-action@v1 for multi arch container build
3. Add the platforms input to the docker/build-push-action@v2 action to target linux/amd64 and linux/arm64

Ideas:

• Allow filter to match case insensitive (-- RENDER -- or -- render -- or -- Render -- all match) Will not implement per #63
• Strip empty lines from displaying (maybe only from edges, not between lines) #68
• Code cleanup, maybe some kind of map and filter to process the signs? #63
• Figure out how to horizontally center sign test within render.
• Figure out how to break the sign filter code into multiple functions (see #68)

Could you please add ability to use custom Overviewer build, and not one fetched from repository. Since Overviewer devs are pushing fixes not so often, this will be very nice feature.

Something like using volume
-v /custom-build:/home/overviewer:ro
Where /custom-build contains overviewer files(overviewer.py etc.)

Found when researching #49, the use of -N in the wget command may not be acting as desired. The intention was to cache the Minecraft Client .jar (would require a volume mapping since this is inside the container).

-O file
...
For this reason, -N (for timestamp-checking) is not supported in combination with -O: since file is always newly created, it will always have a very new timestamp. A warning will be issued if this combination is used.

Also, it wouldn't hurt to have some documentation around how to use this cache.

Need to confirm that the "Latest" tag logic works, as it downloads to latest.jar and Overviewer may not look there.

Hello.
It seems that it's not possible to set this up for Bedrock Servers, even though Overviewer works with Bedrock maps.

The big issue is the path it is looking at, assuming that the level data is directly in home/minecraft/server/worlds.
When bind-mounting a Bedrock Server to the server directory, the actual level data is located in /home/minecraft/server/worlds/[LevelName].

Hello! Thank you for this project.
An error occurs since 21 days during the third Docker building step due to an unknown version of apt package python3-pil.

You can see full error on GitHub actions of your project:
https://github.com/mide/minecraft-overviewer/runs/5000158213?check_suite_focus=true

Log:

Reading package lists...
Building dependency tree...
Reading state information...
E: Version '4.0.0-4+deb9u3' for 'python3-pil' was not found

It seems that blue torches are not being rendered. That is all. Thanks.

Hey! This looks like a pretty fun idea. Is there a guide or docs anywhere for setting this up with an existing Minecraft server running on docker compose? I.e., getting the filesystem working well

Log:

2018-04-22T17:27:59.024590033Z 2018-04-22 17:27:59 (670 KB/s) - '/home/minecraft/.minecraft/versions/1.12.2/1.12.2.jar' saved [10180113/10180113]
2018-04-22T17:27:59.024603027Z
2018-04-22T17:27:59.152183651Z 2018-04-22 17:27:59  Welcome to Minecraft Overviewer!
2018-04-22T17:27:59.153658711Z *******************************************************************************
2018-04-22T17:27:59.153688077Z 2018-04-22 17:27:59 E An error has occurred. This may be a bug. Please let us know!
2018-04-22T17:27:59.153701987Z See http://docs.overviewer.org/en/latest/index.html#help
2018-04-22T17:27:59.153708137Z
2018-04-22T17:27:59.153713429Z This is the error that occurred:
2018-04-22T17:27:59.153719518Z Traceback (most recent call last):
2018-04-22T17:27:59.153724837Z   File "/usr/bin/overviewer.py", line 624, in <module>
2018-04-22T17:27:59.153730905Z     ret = main()
2018-04-22T17:27:59.153736217Z   File "/usr/bin/overviewer.py", line 427, in main
2018-04-22T17:27:59.153741710Z     assetMrg = assetmanager.AssetManager(destdir, config.get('customwebassets', None))
2018-04-22T17:27:59.153747293Z   File "/usr/lib/python2.7/dist-packages/overviewer_core/assetmanager.py", line 47, in __init__
2018-04-22T17:27:59.153752930Z     self.fs_caps = get_fs_caps(self.outputdir)
2018-04-22T17:27:59.153758513Z   File "/usr/lib/python2.7/dist-packages/overviewer_core/files.py", line 27, in get_fs_caps
2018-04-22T17:27:59.153774017Z     return {"chmod_works": does_chmod_work(dir_to_test),
2018-04-22T17:27:59.153793604Z   File "/usr/lib/python2.7/dist-packages/overviewer_core/files.py", line 38, in does_chmod_work
2018-04-22T17:27:59.153799338Z     f1 = tempfile.NamedTemporaryFile(dir=dir_to_test)
2018-04-22T17:27:59.153804671Z   File "/usr/lib/python2.7/tempfile.py", line 475, in NamedTemporaryFile
2018-04-22T17:27:59.153810212Z     (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags)
2018-04-22T17:27:59.153815512Z   File "/usr/lib/python2.7/tempfile.py", line 244, in _mkstemp_inner
2018-04-22T17:27:59.153820980Z     fd = _os.open(file, flags, 0600)
2018-04-22T17:27:59.153826316Z OSError: [Errno 13] Permission denied: '/home/minecraft/render/tmp8aeEXv'

Command:

docker run -v /mc/data/worldNew:/home/minecraft/server/world/:ro -v /mc/overviewer/worldNew:/home/minecraft/render/:rw -e MINECRAFT_VERSION=1.12.2 mide/minecraft-overviewer:latest

I was getting minecraft-overviewer set up on on a new machine. I thought about building a docker image but wondered if a good soul had done the work already. They had ... and then I looked at the author, hi Mark!

Not an issue, just a thank you.

The 1.13.2.jar downloads but then overviewer doesn't find it

$ docker run   --rm   -e MINECRAFT_VERSION="1.13.2"   -v /Users/[me]/Desktop/[minecraft_server]:/home/minecraft/server/:ro   -v /Users/[me]/Desktop/[minecraft_server_map]:/home/minecraft/render/:rw   mide/minecraft-overviewer:latest
Using Client URL https://launcher.mojang.com/v1/objects/30bfe37a8db404db11c7edf02cb5165817afb4d9/client.jar.
WARNING: timestamping does nothing in combination with -O. See the manual
for details.

--2019-02-28 01:28:01--  https://launcher.mojang.com/v1/objects/30bfe37a8db404db11c7edf02cb5165817afb4d9/client.jar
Resolving launcher.mojang.com (launcher.mojang.com)... 54.192.147.10
Connecting to launcher.mojang.com (launcher.mojang.com)|54.192.147.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 16089640 (15M) [application/java-archive]
Saving to: '1.13.2.jar'

     0K .......... .......... .......... .......... ..........  0% 2.79M 5s
[spliced]
 15700K .......... ..                                         100% 8.04M=4.6s

2019-02-28 01:28:05 (3.31 MB/s) - '1.13.2.jar' saved [16089640/16089640]

2019-02-28 01:28:06  Welcome to Minecraft Overviewer!
2019-02-28 01:28:15  Generating textures...
*******************************************************************************
2019-02-28 01:28:15 E Could not find the textures while searching for 'assets/minecraft/textures/block/grass_block_top.png'. Try specifying the 'texturepath' option in your config file.
Set it to the path to a Minecraft Resource pack.
Alternately, install the Minecraft client (which includes textures)
Also see <http://docs.overviewer.org/en/latest/running/#installing-the-textures>
(Remember, this version of Overviewer requires a 1.13-compatible resource pack)
(Also note that I won't automatically use snapshots; you'll have to use the texturepath option to use a snapshot jar)

It'd be good to run shellcheck and flake8 to ensure formatting is optimal. Perhaps even a Dockerfile linter or a Markdown linter.

Per the Overviewer documentation, text retrieved from Minecraft signs must be escaped to prevent HTML and/or JavaScript injection attacks. The default configuration which ships with this project permits players to inject arbitrary HTML tags into the Leaflet page. The injected HTML will become part of the DOM when when the user interacts with the sign marker. On click, it will be rendered into the leaflet-popup-pane div.

Per the Minecraft wiki, each sign contains four text fields named Text1..Text4. Each line is a TAG_String. Tags of this type can individually accommodate up to 2^16 bytes of JSON-wrapped text. This is more than enough space for a program of great complexity.

The normal Minecraft client does not permit unfettered access to the TAG_String's entire storage space: it places a very low limit on the length of each line. This, as well as the Docker image's default behavior of inserting <br/> tags between lines, makes it difficult to inject JavaScript via a normal Minecraft client. With that said, it is easy to gain access to the NBT tag's entire storage space via command blocks or Op commands.

The following proof-of-concept exploit, tested on Minecraft v1.16.5, requires Op-level access:

/setblock ~1 ~ ~ minecraft:acacia_sign replace
/data modify block ~1 ~ ~ Text1 set value '{"text":"<style onload=\\"alert(\'test\');\\"/>"}'

When rendered to the Overviewer Map, this sign will popup a JavaScript alert when clicked.

The proof-of-concept exploit is not specific to v1.16.5 and can probably be adapted for many other versions of Minecraft. This particular set of commands requires Op permission, but I suspect that a modded client may be able to convince the server to store more sign text than normal. I have not tested this. A quick search suggests that the vanilla client limits sign line length based on resource pack. Resource packs are trivial to modify locally. I am not sure what limits the server itself enforces, if any.

To inject a script or other content onto an Overviewer page,

1. Overviewer must be configured to extract and publish Minecraft signs as markers. (This is a typical configuration.)
2. An authorized user account on the Minecraft server must create a sign that includes script or HTML content.
3. The victim must then explicitly interact with the generated sign marker. Unless otherwise mitigated by a Content Security Policy, any injected JavaScript will run in the Overviewer/Leaflet's script security context.

Fix for the Docker Image (this project)

The upstream project recommends using python's html.escape() method to escape characters of special significance to HTML. I recommend applying a similar fix here.

Fix for the Upstream Project

The need to escape text stems from Overviewer's use of the Leaflet API,

overviewer_core/data/js_src/util.js

if (marker_entry.createInfoWindow && db.text) {
    layerObj.bindPopup(db.text);
}

which eventually invokes the Leaflet method with content = db.text,

src/layer/DivOverlay.js

if (typeof content === 'string') {
    node.innerHTML = content; // <--- injection vulnerability
} else {
    while (node.hasChildNodes()) {
        node.removeChild(node.firstChild);
    }
    node.appendChild(content);
}

A safer alternative would be to provide a complete DOM node as input here, which is officially documented in Leaflet. A text node would work just fine:

if (marker_entry.createInfoWindow && db.text) {
    layerObj.bindPopup(document.createTextNode(db.text));
}

Here, the popup will contain the literal text db.text without the need to escape anything. Preserving the original line-by-line sign layout could be accomplished with a CSS white-space: pre; property.

EDIT: for proof-of-concept and proposed fix for upstream project.

/usr/bin/overviewer.py --genpoi: error: unrecognized arguments: --no-tile-checks

Looks like the first phase of scanning works correctly but the extra arguments stop POI generation from working.

I'm trying to run a fresh container using docker compose:

version: '2'
services:
  minecraft-overviewer-app:
    image: mide/minecraft-overviewer:latest
    container_name: minecraft-overviewer-app
    environment:
      - MINECRAFT_VERSION="latest"

    volumes:
      - /var/lib/docker/volumes/minecraft_data/_data/:/home/minecraft/server/:ro
      - ./data/map/:/home/minecraft/render/:rw
      - ./data/minecraft/:/home/minecraft/.minecraft/:rw

  minecraft-overviewer-nginx:
    image: docker.io/bitnami/nginx:latest
    container_name: minecraft-overviewer-nginx
    restart: unless-stopped
    volumes:
      - ./data/map/:/app
    ports:
      - '82:8080'
    restart: unless-stopped
    networks:
      - proxy
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.overviewer.entrypoints=http"
      - "traefik.http.routers.overviewer.rule=Host(`example.com`)"
      - "traefik.http.middlewares.overviewer-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.routers.overviewer.middlewares=overviewer-https-redirect"
      - "traefik.http.routers.overviewer-secure.entrypoints=https"
      - "traefik.http.routers.overviewer-secure.rule=Host(`example.com`)"
      - "traefik.http.routers.overviewer-secure.tls=true"
      - "traefik.http.routers.overviewer-secure.service=overviewer"
      - "traefik.http.services.overviewer.loadbalancer.server.port=8080"
      - "traefik.docker.network=proxy"

networks:
  proxy:
    external: true

But the container stops with the following error in logs:

# docker logs minecraft-overviewer-app
Traceback (most recent call last):
  File "/home/minecraft/download_url.py", line 52, in <module>
    print(get_minecraft_download_url(args.version))
  File "/home/minecraft/download_url.py", line 30, in get_minecraft_download_url
    version, MANIFEST_URL
RuntimeError: Couldn't find Minecraft Version "latest" in manifest file https://launchermeta.mojang.com/mc/game/version_manifest.json.

Using version numbers, for example 1.18.2 doesn't work either, it throws the same exception.

When running in a 1.17 env it returns

(Remember, this version of Overviewer requires a 1.16-compatible resource pack)
(Also note that I won't automatically use snapshots; you'll have to use the texturepath option to use a snapshot jar

It would be nice to render signs on the map without always having to see -- RENDER -- at the end of them.

I'll work on a PR for this tomorrow probably.

Check if we can upgrade the wrapper scripts and the general runtime to Python3.

E.G Update entrypoint.sh to include:

FILE=/home/minecraft/config/config.py
if [ -f "$FILE" ]; then
    CONFIG = /home/minecraft/config/config.py
else 
    CONFIG = /home/minecraft/config.py
fi

And change the overviewer commands to
--config $CONFIG

I think this should allow custom config.py files by mounting a volume at /home/minecraft/config/ whist still defaulting to the included one if there's not one there.
Possibly there's a better way to do it that I'm missing.

When running the docker image with the following run statement:

docker run \
  --rm \
  -e MINECRAFT_VERSION="1.15.2" \
  -v /opt/minecaft/compatibility_issues:/home/minecraft/server/:ro \
  -v /opt/minecraft/overviewer:/home/minecraft/render/:rw \
  mide/minecraft-overviewer:latest

I get the following error:

2020-01-26 20:23:42  Welcome to Minecraft Overviewer!
*******************************************************************************
2020-01-26 20:23:42 E An error has occurred. This may be a bug. Please let us know!
See http://docs.overviewer.org/en/latest/index.html#help

This is the error that occurred:
Traceback (most recent call last):
  File "/usr/bin/overviewer.py", line 650, in <module>
    ret = main()
  File "/usr/bin/overviewer.py", line 443, in main
    assetMrg = assetmanager.AssetManager(destdir, config.get('customwebassets', None))
  File "/usr/lib/python3.5/dist-packages/overviewer_core/assetmanager.py", line 47, in __init__
    self.fs_caps = get_fs_caps(self.outputdir)
  File "/usr/lib/python3.5/dist-packages/overviewer_core/files.py", line 27, in get_fs_caps
    return {"chmod_works": does_chmod_work(dir_to_test),
  File "/usr/lib/python3.5/dist-packages/overviewer_core/files.py", line 38, in does_chmod_work
    f1 = tempfile.NamedTemporaryFile(dir=dir_to_test)
  File "/usr/lib/python3.5/tempfile.py", line 688, in NamedTemporaryFile
    (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
  File "/usr/lib/python3.5/tempfile.py", line 399, in _mkstemp_inner
    fd = _os.open(file, flags, 0o600)
PermissionError: [Errno 13] Permission denied: '/home/minecraft/render/tmpwqwyymrh'```

When started, the container exits with error:
/home/minecraft/entrypoint.sh: line 11: python: command not found,

Image: mide/minecraft-overviewer:latest

I keep getting this error, i don´t know whats wrong

This is the error that occurred:
Traceback (most recent call last):
File "overviewer.py", line 653, in
File "overviewer.py", line 578, in main
File "overviewer_core\assetmanager.py", line 76, in initialize
File "overviewer_core\assetmanager.py", line 150, in _output_assets
File "PIL\Image.py", line 1932, in save
PermissionError: [Errno 13] Permission denied: 'C:\Users\User\Desktop\map\world-lighting\blank.png'

The rendering software that this docker image makes use of (Minecraft Overviewer) doesn't yet support the map format released during 1.13. See overviewer/Minecraft-Overviewer#1454 for information on their progress.

This container should update within a day due to the automated build, but I'll keep tabs on it, too.