l0laapk3 / factoriomaps Goto Github PK

View Code? Open in Web Editor NEW

This project forked from jahands/factoriomods_factoriomaps

120.0 3.0 22.0 4.41 MB

L0laapk3's FactorioMaps mod

Home Page: https://mods.factorio.com/mod/L0laapk3_FactorioMaps

License: Other

Lua 29.75% Python 53.41% CSS 3.03% HTML 0.65% JavaScript 13.17%

factorio factorio-mod factorio-data timelapse timeline browser hacktoberfest

factoriomaps's Introduction

Factorio Maps

This Factorio mod turns your factory into a timeline! You can view the map locally or upload it to a web server.

Mod portal link: https://mods.factorio.com/mod/L0laapk3_FactorioMaps

How to Install

Note that this program now only runs on 64 bit python version 3.6 or higher.

Download FactorioMaps to %appdata%\mods\, either from the mod portal (The mod does not need to be enabled to work) and then unzipping it, or from the github releases page.
Install the latest version of 64 bit python 3. (Do not install python 2.) Make sure to do a select the "add python to PATH" and "install pip" options.
Inside the factoriomaps folder, install the required pip packages: python -m pip install --upgrade -r requirements.txt.

How to Use

Make sure you close factorio before starting the process.
Navigate to the FactorioMaps folder (%appdata%\Factorio\mods\FactorioMaps_x.x.x). Unzip it if you haven't done that already.
Open a command line by typing cmd in the address bar and pressing enter.
Run python auto.py. Some syntax examples:
- python auto.py Generate a snapshot of the latest modified map (autosaves are excluded) and store it to a folder with the same name. If the folder already exists, the snapshot will be appended to the timeline.
- python auto.py savename Generate a snapshot of savename and store it to folder savename.
- python auto.py outfolder savename Generate a snapshot of savename and store it to folder outfolder.
- python auto.py outfolder savename1 savename2 savename3 Generate timeline snapshots of savename1, savename2, savename3 in that order, and store it to folder outfolder.
- python auto.py outfolder savename* Generate timeline snapshots of all savefiles that match the glob pattern savename* in natural order, and store it to folder outfolder.
- python auto.py --factorio=PATH Same as python auto.py, but will use factorio.exe from PATH instead of attempting to find it in common locations.
- python auto.py --verbose Displays factoriomaps related logs.
- python auto.py --verbosegame Displays all game logs.
- python auto.py --basepath=PATH Same as python auto.py, but will output to PATH instead of script-output\FactorioMaps. Not recommended to use.
An index.html will be created in %appdata%\Factorio\script-output\FactorioMaps\mapName. Enjoy!

Configuration

Heres a list of flags that auto.py can accept: Options with a * do not have an effect when appending to existing timelapses.

flag	Description
`--dayonly`*	Do not take nighttime screenshots (For now, this setting needs to be the same across one timeline).
`--nightonly`*	Do not take daytime screenshots.
`--hd`*	Take screenshots of resolution 64 x 64 pixels per in-game tile instead of 32 x 32 to match the resolution of the newer HD textures.
`--no-altmode`	Hides entity info (alt mode)
`--no-tags`	Hides map tags
`--default-timestamp=-1`	Snapshot that will be loaded by the webpage by default. Negative values indicate newest snapshots, so -1 indicates the newest map while 0 indicates the oldest map.
`--build-range=5.2`*	The maximum range from buildings around which pictures are saved (in chunks, 32 by 32 in-game tiles).
`--connect-range=1.2`*	The maximum range from connection buildings (rails, electric poles) around which pictures are saved.
`--tag-range=5.2`*	The maximum range from mapview tags around which pictures are saved.
`--surface=nauvis`	Used to capture other surfaces. If left empty, the surface the player is standing on will be used. To capture multiple surfaces, use the argument multiple times: `--surface=nauvis --surface="Factory floor 1"`. To find out the names of surfaces, use the command `/c for _,s in pairs(game.surfaces) do game.print(s.name) end`.
`--factorio=PATH`	Use `factorio.exe` from PATH instead of attempting to find it in common locations.
`--modpath=PATH`	Use PATH as the mod folder.
`--basepath=RELPATH`	Output to `script-output\RELPATH` instead of `script-output\FactorioMaps`. (Factorio cannot output outside of `script-output`)
`--date=dd/mm/yy`	Date attached to the snapshot, default is today.
`--steam`	Only use factorio binary from steam.
`--standalone`	Only use standalone factorio binary.
`--verbose`	Displays factoriomaps script logs.
`--verbosegame`	Displays all game logs.
`--no-update`	Skips the update check.
`--maxthreads=N`	Sets the number of threads used for all steps. By default this is equal to the amount of logical processor cores available.
`--cropthreads=N`	Sets the number of threads used for the crop step.
`--refthreads=N`	Sets the number of threads used for the crossreferencing step.
`--zoomthreads=N`	Sets the number of threads used for the zoom step.
`--screenshotthreads=N`	Set the number of screenshotting threads factorio uses.
`--delete`	Deletes the output folder specified before running the script.
`--dry`	Skips starting factorio, making screenshots and doing the main steps, only execute setting up and finishing of script.
`--force-lib-update`	Forces an update of the web dependencies.
`--temp-dir`	Use a custom temporary directory.

Image quality settings can be changed in the top of zoom.py.

Result folder estimates

You can expect the resulting folders to take up approx. (very rough estimate) 15 times the savefile size per timestamp per daytime for day images and 10 times for night images. The intermediate total disk usage will be much higher, 10 times the final result or more. If this is a problem for you, go put a +1 on #46. Of course the processing time depends very heavely on your system specs, but a rough estimate is an hour per timestamp per daytime per 50 MB of savefile.

Hosting this on a server

If you wish to host your map for other people to a server, you need to take into account the following considerations: (You can change these once in index.html.template and they will be used for all future snapshots.)

Of the files that this program generates, the files required to be hosted are:
- index.html
- index.css
- index.js
- mapInfo.js
- All images in Images\.
- All files in lib\. All other files, including txt and other non-image files in Images\, are not used by the client. Some of them are temporary files, some of them are used as savestate to create additional snapshots on the timeline.

Known mods that make use of the API to improve compability

* Factorissimo ⩾2.3.5: Able to render the inside of factory buildings recursively.
* Your mod? If you want to have a chat, you can always find me on discord: L0laapk3#2010

Known limitations

If you only have the steam version of factorio, steam will ask you to confirm the arguments everytime the script tries to start up. The popup window will sometimes not focus properly. Please press alt tab a couple of times until it shows up. The only way to get around this is to install the standalone version of factorio.
If the program crashes while making a snapshot, it is very likely to leave timelines behind in a 'bricked' state and will probably mess up future snapshots. The easiest way is to simply start over and regenerate all the snapshots from old savefiles. If thats not a possibility, feel free to contact me on discord (L0laapk3#2010) or create an Issue, I'll do my best to help you out.
Running this on headless servers is not possible due to factorio limitations.
Not all factorio rich text is supported #93.

Issues

If you have problems or questions setting things up, feel free to reach out to me on discord at L0laapk3#2010. If you believe you have found a bug, inconsistency, something unclear or anything else, please try generating a map to a new empty output folder (If you need help recovering bricked timelapses, please reach out to me). If the problem persists, please submit an issue to the Issue tracker.

factoriomaps's People

Contributors

Stargazers

Watchers

factoriomaps's Issues

Remove jQuery and use HTML5's slider instead of JS driven sliders

Just a feature enhancement to reduce the javascript processing (especially on mobile) for the generated output. :-)

[Suggestion] Remove smoke

I do not use the mod yet, but in the live demo I noticed that the smoke is in the map.
You can not do anything with Google Maps, but at Factorio it would be better without smoke.

https://i.imgur.com/3jSIErM.jpg

Big Map Processing Not Working (Maybe map Lua?)

Hello,
I am trying to create a FactorioMaps preview for this actual map : https://factorio-saves.poli.fun/Home/Join-And-Play-Coop-Saves/Old-Saves/06-Feb-12:54.zip

Sadly i the map don't get corped i except 2 things :

The Map has blank spaces between discorvered one (everything is not bounded)
The Lua's script in the map make some problems

Is there any fix for maps like this maybe could you try to fix it by your side ?
Thanks Poli ^^

Problem still persists but different error this time around.

__L0laapk3_FactorioMaps__/autorun.lua:9:
    '}' expected near '='

Originally posted by @crazyjoery in #5 (comment)

How to use with old factorio maps?

I have a bunch of old maps that I saved with earlier versions of FactorioMaps. They used the Google Maps API and now don't work (even if I put in a valid API key).

It appears that the new version doesn't use Google Maps but I'm guessing still uses the same image format and layout. Is there a way to generate a new index and library files and mapInfo.js for an old map?

Note: many of these maps were from factorio versions and mod packages that would be hard to reproduce so I can't use the new process and go from save to map.

Mod crashes on loading

Factorio itself crashes with the Lua script with the error:

[GAME]  174.753 Factorio initialised                                                                                                                                                                       
[GAME]  174.754 Mods to disable:Failed to load mods: __L0laapk3_FactorioMaps__/data-final-fixes.lua:35: attempt to perform arithmetic on field 'r' (a nil value)                                           
[GAME] stack traceback:                                                                                                                                                                                    
[GAME]  __L0laapk3_FactorioMaps__/data-final-fixes.lua:35: in function 'index'                                                                                                                             
[GAME]  __L0laapk3_FactorioMaps__/data-final-fixes.lua:59: in main chunk                                                                                                                                   
[GAME] Mods to be disabled:                                                                                                                                                                                
[GAME] • L0laapk3_FactorioMaps

Error when copying icons to webroot

When executing auto.py i get an error after the cleanup step:

cleaning up
Traceback (most recent call last):
  File "./auto.py", line 702, in <module>
    auto(*sys.argv[1:])
  File "./auto.py", line 628, in auto
    copy(src, dest)
  File "/home/myusername/.local/share/virtualenvs/L0laapk3_FactorioMaps_3.4.0-niQMlvrd/lib/python3.7/shutil.py", line 245, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/home/myusername/.local/share/virtualenvs/L0laapk3_FactorioMaps_3.4.0-niQMlvrd/lib/python3.7/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:
NotADirectoryError: [Errno 20] Not a directory: '/home/myusername/Downloads/factorio/bin/x64/factorio/../../../data/base/graphics/icons/electronic-circuit.png'

This is because bin/x64/factorio is not a directory, it is an executable. I made the following change:

diff --git a/auto.py b/auto.py
index 9a22f87..763162b 100644
--- a/auto.py
+++ b/auto.py
@@ -607,7 +607,7 @@ def auto(*args):
                                iconColor = m.group(2).split("?")
                                icon = iconColor[0]
                                if m.group(1) in ("base", "core"):
-                                       src = os.path.join(factorioPath, "../../../data", m.group(1), icon + ".png")
+                                       src = os.path.join(os.path.split(factorioPath)[0], "../../data", m.group(1), icon + ".png")
                                else:
                                        mod = next(mod for mod in modVersions if mod[0] == m.group(1).lower())
                                        if not mod[1][3]: #true if mod is zip

kwarg checking

Multi surface support

The ability to switch between surfaces in browser.

Other Surfaces support

Would it be possible to generate the chunks for all loaded surfaces as well? It would be useful to see Factorissimo buildings as well as a few others that this pack has.

Mixed day/night settings in timelines

Map label generation crash

Hi,

Recently tried to use this code to generate a map for a seablocks game i've been playing.
Managed to get the map gen to work, however the code is dying when trying to generate labels

generating mapInfo.json
updating labels
disabling FactorioMaps mod
cleaning up
Traceback (most recent call last):
  File "auto.py", line 626, in <module>
    auto(*sys.argv[1:])
  File "auto.py", line 545, in auto
    mod = next(mod for mod in modVersions if mod[0] == m.group(1).lower())
StopIteration

For now I've removed the code in auto.py that handles the tag / label gen, and re-ran the gen with --dry to produce the mapinfo.js and other files required to view the map.

If i re-run the script with --dry without removing the label gen stuff i get the following

$ python auto.py --factorio=/e/factorio_maps_gen/bin/x64/factorio.exe --verbosegame --dayonly --dry
No save name passed. Using most recent save: SeaBlocks0.16.5
factorio path: e:\factorio_maps_gen\bin\x64\factorio.exe
output folder: script-output\FactorioMaps\SeaBlocks0.16.5
checking for updates
enabling FactorioMaps mod
updating labels
Traceback (most recent call last):
  File "auto.py", line 529, in auto
    rawPath = rawTags[tag["iconType"] + tag["iconName"][0].upper() + tag["iconName"][1:]]
  File "<string>", line 2, in __getitem__
  File "C:\Users\User\AppData\Local\Programs\Python\Python37\lib\multiprocessing\managers.py", line 811, in _callmethod
    raise convert_to_error(kind, result)
KeyError: 'fluidLiquid-molten-nickel'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "auto.py", line 619, in <module>
    auto(*sys.argv[1:])
  File "auto.py", line 594, in auto
    kill(pid)

Happy to upload a copy of the map and mods if you need that for testing. I do use a fair few mods which might be the cause of this!

Symlinks from script-output do not work in 0.17

If you would like this to work again, please redirect your concerns to this factorio thread:

https://forums.factorio.com/viewtopic.php?f=7&t=68499

Crash on run

Started up our new 0.17 modded server and tried to run the script, crash on first run, very quickly:

╰─➤  python auto.py --verbosegame OverGreg3 OverGreg3-0
OverGreg3
OverGreg3-0
factorio path: /mnt/storage0/overminddl1/Games/Factorio/factorio_0.17.16/bin/x64/factorio
output folder: script-output/FactorioMaps/OverGreg3
downloading https://cdn.jsdelivr.net/npm/leaflet@1.4.0/dist/leaflet.css
downloading https://cdn.jsdelivr.net/npm/leaflet@1.4.0/dist/leaflet-src.min.js
downloading https://cdn.jsdelivr.net/npm/leaflet.fullscreen@1.4.5/Control.FullScreen.css
downloading https://cdn.jsdelivr.net/npm/leaflet.fullscreen@1.4.5/Control.FullScreen.min.js
downloading https://cdn.jsdelivr.net/npm/jquery@3.3.1/dist/jquery.min.js
downloading https://ajax.googleapis.com/ajax/libs/jqueryui/1.12.1/themes/smoothness/jquery-ui.css
downloading https://ajax.googleapis.com/ajax/libs/jqueryui/1.12.1/jquery-ui.min.js
downloading https://cdn.jsdelivr.net/gh/L0laapk3/Leaflet.OpacityControls@1/Control.Opacity.css
downloading https://cdn.jsdelivr.net/gh/L0laapk3/Leaflet.OpacityControls@1/Control.Opacity.js
downloading https://factorio.com/static/img/favicon.ico
downloading https://i.factoriomaps.com/t92kw.png
enabling FactorioMaps mod
cleaning up  
building autorun.lua                                                                                                                                                                                          
starting factorio                                                                                                                                                                                             
ln: missing file operand
Try 'ln --help' for more information.
Traceback (most recent call last):
  File "auto.py", line 376, in auto
    linkDir(os.path.join(tmpDir, "script-output"), "../../script-output")
  File "auto.py", line 252, in linkDir
    subprocess.check_call(cmd, stdout=subprocess.DEVNULL, shell=True)
  File "/home/overminddl1/.asdf/installs/python/3.7.2/lib/python3.7/subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('ln', '-s', '/tmp/FactorioMaps-319988162/script-output', '/mnt/storage0/overminddl1/Games/Factorio/factorio_0.17.16/script-output')' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "auto.py", line 629, in <module>
    auto(*sys.argv[1:])
  File "auto.py", line 604, in auto
    kill(pid)
UnboundLocalError: local variable 'pid' referenced before assignment

Two things, first is that absolutely no symlink should be getting created to to my /tmp partition, especially nothing as massive as FactorioMaps as the tmp partition is waaaaaay too small for something as huge as that (that is why I have it generating on an external drive directly as it is, the /mnt/storage0 bit), and second, something like this should absolutely not be touching anything outside of the factorio directory without explicit consent.

map tag crash on steam version

Trying to run auto.py on this map in Factorio 16.5.1 via Steam causes the following error to occur:

Traceback (most recent call last):  
  File "auto.py", line 559, in <module>  
    auto(*sys.argv[1:])  
  File "auto.py", line 464, in auto  
    rawPath = rawTags[tag["iconType"] + tag["iconName"][0].upper() + tag["iconName"][1:]]  
KeyError: 'itemIron-plate'

The map only had QoL mods and nothing that added new items, still tried disabling all mods and trying again, no change.

Full run log:

python auto.py --dayonly --hd
No save name passed. Using most recent save: trunklunk
factorio path: C:\Program Files (x86)\Steam\steamapps\common\Factorio\bin\x64\factorio.exe
output folder: script-output\FactorioMaps\trunklunk
checking for updates
enabling FactorioMaps mod
cleaning up
building autorun.lua
starting factorio
WARNING: running in limited support mode trough steam. Consider using standalone factorio instead.
         Please open steam and confirm the steam 'start game with arguments' popup.
Processing trunklunk/45/nauvis/day (1 of 1)
crop 100.0% [=========================================================================================================]
killed factorio
zoom 100.0% [=========================================================================================================]
generating mapInfo.json
updating labels
disabling FactorioMaps mod
cleaning up
Traceback (most recent call last):
  File "auto.py", line 559, in <module>
    auto(*sys.argv[1:])
  File "auto.py", line 464, in auto
    rawPath = rawTags[tag["iconType"] + tag["iconName"][0].upper() + tag["iconName"][1:]]
KeyError: 'itemIron-plate'

Map markers with no icon set display as broken images

In my map I made several map markers as temporary TODOs to myself, and didn't give them icons. They show up like this after processing:

The generated HTML looks like this:

<img class="map-marker" src="undefined">
<span class="map-marker-text">todo: Milsci, lasers, then iron mine</span>

The broken image from the undefined src looks pretty bad.

Some thoughts on solutions:

Hide images with undefined srcs: img.map-marker[src="undefined"] { display: none; }
Don't generate the img element if the src is undefined
Use a simple dot image, maybe via SVG data URI or by adding a small marker image.

0.17 todo list

how do you make a timeline?

I managed to get this running for my savegames but I'm not getting a nice timeline like the linked example, only a snapshot of the current version. how do you get to see the factory evolution trough time?

Game not outputting log properly

When the script launches factorio the output of factorio is being placed on a single line that is constantly overwriting itself, thus making it very difficult to see what is happening. It seems the change happened in this commit:
6d0cda0

error when script-output not present

New install 0.17. Nothing special. Download and install as describe.

When I run auto, it gets errors about not being to find specified directory.

After I create "script-output" by hand it works fine.

Factorio isn't killed on posix systems

It seems the process kill call is incorrect.

Loading a save game that's saved while the current player is on another surface freezes the script

I loaded a save while inside a factorissimo building and after printing the last line from the factorio program output it just never did anything else after it printed this:

Processing GregOver2/19/Factory/floor/8/day (1 of 2)

Nothing ever happened. Had to load the save, exit the building, then save again and run it.

psutil access denied

Traceback (most recent call last):
  File "auto.py", line 339, in <module>
    auto(*sys.argv[1:])
  File "auto.py", line 250, in auto
    crop(outFolder, otherInputs[0], otherInputs[1], otherInputs[2], basepath)
  File "/home/overminddl1/Games/Factorio/tmp/mods/L0laapk3_FactorioMaps_2.4.2/crop.py", line 29, in crop
    psutil.Process(os.getpid()).nice(psutil.BELOW_NORMAL_PRIORITY_CLASS if os.name == 'nt' else -10)
  File "/home/overminddl1/.local/lib/python3.6/site-packages/psutil/__init__.py", line 716, in nice
    self._proc.nice_set(value)
  File "/home/overminddl1/.local/lib/python3.6/site-packages/psutil/_pslinux.py", line 1513, in wrapper
    raise AccessDenied(self.pid, self._name)
psutil._exceptions.AccessDenied: psutil.AccessDenied (pid=20832)

Wrapping the psutil calls with an exception handler and ignoring it (since it is only a nice thing and not anything required) fixes it.

However, the call itself is inherently wrong, right now it is:

psutil.Process(os.getpid()).nice(psutil.BELOW_NORMAL_PRIORITY_CLASS if os.name == 'nt' else -10)

-10 will give it more CPU power (less being 'nice' as per the name of the call), and to set a process to be less nice than 0 (default) requires having root permissions. I'm wondering if this was actually supposed to be 10 instead of -10 considering the windows version is set to BELOW_NORMAL_PRIORITY_CALL. Thus I ended up changing all the nice -10 values to 10 and that worked as well without needing an exception handler.

Crash on run

The auto.py crash after the image are generated. No index.html and related program can be found in script-output.

Traceback (most recent call last):
File "auto.py", line 629, in
auto(*sys.argv[1:])
File "auto.py", line 568, in auto
newImg = ImageChops.multiply(newImg, Image.new("RGBA", img.size, color=tuple(map(lambda s: int(round(float(s))), iconColor[1].split("%")))))
AttributeError: 'NoneType' object has no attribute 'size'

Python version: 3.7.3
OS: Windows 10
Game version: 0.17.29

Crop freezes on fresh map

I tend to make a save at the very start of a map before having even moved and name it something like GregOver1a (next one would be GregOver1b for example), however running the generator on that map seems to generate no images, and thus when it gets to the Cropping stage of the python script it freezes there forever.

Error when trying to regen map images

When trying to regenerate the images from a active save that has been generated before it throws a error during the loading stages of the save.

    __L0laapk3_FactorioMaps__/autorun.lua:9:
    '}' expected near ':'

Here is the code from the autorun.lua file

fm.autorun = {
    name = "No_end/",				-- changing this is currently not supported
    day = true,			 			-- changing this is currently not supported
    night = true,		 			-- changing this is currently not supported
    around_build_range = 5.2,		-- max range from buildings that images will be saved. Feel free to crank this up to very large numbers, it will only render chunks that already exist, it will not generate new ones.
    around_smaller_range = 1,		-- same as above, but smaller range for the following entity types:
    smaller_types = {"lamp", "electric-pole", "radar", "straight-rail", "curved-rail", "rail-signal", "rail-chain-signal", "locomotive", "cargo-wagon", "fluid-wagon", "car"},
    date = "07/10/18",
    mapInfo = {seed =1069981988,maps ={{tick =3395,path ="1",date ="07/10/18",mods ={AGoodStart ="0.1.0","Additional-Turret-016":"0.2.1",AlternativeSteam ="1.0.2","Armored-train":"0.1.11",AtomicArtillery ="0.1.5",AutoDeconstruct ="0.1.11",Automatic_Coupler_016 ="1.0.4",Big_Brother ="0.4.6",Big_Brother_Bobs_plugin ="0.1.7",Bio_Industries ="2.6.12",Blueprint_Flip_Turn ="1.0.0","BurnerLeech A16 Fix":"0.1.8",BurnysTSBC ="0.1.3","Clowns-AngelBob-Nuclear":"1.0.1","Clowns-Extended-Minerals":"1.0.10","Clowns-Locale":"1.0.2","Clowns-Nuclear":"1.2.0","Clowns-Processing":"1.2.8",Crafting_Speed_Research ="0.2.0",Dirt_Path_Redux ="1.2.1",EarlyTrains ="0.16.0",EvoGUI ="0.4.302","Explosive Termites":"1.1.11",FARL ="2.1.2",FNEI ="0.1.9","FactorioExtended-Core":"0.3.1","FactorioExtended-Equipment":"0.3.1","FactorioExtended-Logistics":"0.3.1","FactorioExtended-Machines":"0.3.1","FactorioExtended-Plus-Core":"0.1.4","FactorioExtended-Plus-Equipment":"0.1.2","FactorioExtended-Plus-Logistics":"0.1.4","FactorioExtended-Plus-Machines":"0.1.4","FactorioExtended-Plus-Power":"0.1.3","FactorioExtended-Plus-Storage":"0.1.4","FactorioExtended-Plus-Transport":"0.1.6","FactorioExtended-Plus-Weaponry":"0.1.5","FactorioExtended-Power":"0.3.1","FactorioExtended-Storage":"0.3.1","FactorioExtended-Trains":"0.3.1","FactorioExtended-Transport":"0.3.1","FactorioExtended-Weaponry":"0.3.1","Flare Stack":"2.1.7",ForceFields2 ="2.1.4",FuelTrainStop ="0.16.2",FusionTrain ="0.16.2",GDIW ="0.16.2",Horns ="1.0.2",IncendiaryMunitions ="0.3.1",InlaidLamps ="0.1.0",L0laapk3_FactorioMaps ="1.1.0","LTN-easier":"0.1.0",LoaderRedux ="1.2.7",LogisticTrainNetwork ="1.8.2",Nanobots ="2.0.7",NapalmArtillery ="0.16.4",NauvisDay ="1.0.148",NuclearArtillery ="0.1.1",PCP ="0.0.9",RealisticDecorationCleanup ="0.16.2",RealisticReactors ="2.0.2","RealisticReactors-utilities":"0.16.4","Reinforced-Walls":"0.16.0",SantasNixieTubeDisplay ="0.1.6",ScanningRadar ="0.2.9",Senpais_Trains ="1.2.1",ShinyAngelGFX ="0.16.8",ShinyBobGFX ="0.16.21",SpaceMod ="0.3.12","Squeak Through":"1.2.2",SteamCracking ="1.0.1","TrainSpeedLimit-016":"1.2.0",UnminableVehicles ="1.0.0",UsefulMapColors ="1.1.10",VehicleSnap ="1.16.0",WarehouseAddons ="0.1.1",WaterWell ="1.0.19",Waterfill_v15 ="0.1.3",ZAdv_example ="1.1.5",ZAdv_ruins ="1.0.1",ZAdventure ="2.16.4","aai-programmable-structures":"0.4.4","aai-programmable-vehicles":"0.4.8","aai-signals":"0.3.3","aai-vehicles-chaingunner":"0.3.1","aai-vehicles-flame-tank":"0.2.1","aai-vehicles-flame-tumbler":"0.3.1","aai-vehicles-hauler":"0.3.1","aai-vehicles-laser-tank":"0.3.1","aai-vehicles-miner":"0.3.5","aai-vehicles-warden":"0.2.1","aai-zones":"0.3.4","angels-petrochem-nitrogen-fix":"0.0.3","angelsaddons-oresilos":"0.4.0","angelsaddons-petrotrain":"0.3.3","angelsaddons-pressuretanks":"0.3.0","angelsaddons-smeltingtrain":"0.1.2","angelsaddons-warehouses":"0.3.0",angelsbioprocessing ="0.5.9",angelsindustries ="0.1.0",angelsinfiniteores ="0.7.3",angelspetrochem ="0.7.12",angelspetrochemNaOH ="16.36.1",angelsrefining ="0.9.14",angelssmelting ="0.4.6",base ="0.16.51","battery-recolor":"0.1.2",beautiful_bridge_railway ="0.1.3","bob-basic-belt-reskin":"0.16.1","bobangels-pipe-fix":"0.16.6",bobassembly ="0.16.1",bobelectronics ="0.16.0",bobenemies ="0.16.0",bobgreenhouse ="0.16.0",bobinserters ="0.16.8",boblibrary ="0.16.6",boblogistics ="0.16.23",bobmining ="0.16.0",bobmods_gfxtweak ="0.16.0",bobmodules ="0.16.0",bobores ="0.16.2",bobplates ="0.16.5",bobpower ="0.16.8",bobrevamp ="0.16.3",bobtech ="0.16.6",bobvehicleequipment ="0.16.2",bobwarfare ="0.16.7","bullet-trails":"0.3.1",cheaper_rail_and_concrete ="1.2.0","concrete-and-stone-016":"0.2.5","even-distribution":"0.2.10","expanded-rocket-payloads":"0.16.5",extendedangels ="0.1.5","folk-shuttle":"0.1.13",helmod ="0.7.11",lightorio ="0.16.4","long-reach-research-upgrade":"0.4.3","multi-product-recipe-details":"0.16.4",nicefill ="0.1.5","ore-eraser":"1.4.1",osha_safety ="0.0.2",platforms ="18.5.2",pushbutton ="0.16.0",railloader ="0.5.1","recursive-blueprints":"0.16.2","research-queue":"1.6.7","rso-mod":"3.8.1",scattergun_turret ="3.8.0",textplates ="0.3.3","upgrade-planner":"1.5.3","walls-block-spitters":"0.3.1",xcompat_bobvehicleequipment ="0.16.2"},surfaces ={nauvis ={spawn ={y =0,x =0},zoom ={min =18,max =20},playerPosition ={y =6.84375,x =77.046875},day ="true",night ="true"}}}}},			-- changing this is not supported
    chunkCache = {
		["3395"] = {nauvis ="-3 -1|-3 0|-3 1|-3 2|-3 3|-1 -5|-2 -3|-1 -4|-1 -3|-2 -2|-2 -1|-1 -2|-1 -1|-2 0|-2 1|-1 0|-1 1|-2 2|-2 3|-1 2|-1 3|-2 4|-2 5|-1 4|-1 5|0 -5|1 -6|1 -5|0 -4|0 -3|1 -4|1 -3|0 -2|0 -1|1 -2|1 -1|0 0|0 1|1 0|1 1|0 2|0 3|1 2|1 3|0 4|0 5|1 4|1 5|0 6|1 6|2 -6|2 -5|3 -6|3 -5|2 -4|2 -3|3 -4|3 -3|2 -2|2 -1|3 -2|3 -1|2 0|2 1|3 0|3 1|2 2|2 3|3 2|3 3|2 4|2 5|3 4|3 5|2 6|3 6|4 -6|4 -5|5 -6|5 -5|4 -4|4 -3|5 -4|5 -3|4 -2|4 -1|5 -2|5 -1|4 0|4 1|5 0|5 1|4 2|4 3|5 2|5 3|4 4|4 5|5 4|5 5|4 6|6 -6|6 -5|7 -5|6 -4|6 -3|7 -4|7 -3|6 -2|6 -1|7 -2|7 -1|6 0|6 1|7 0|7 1|6 2|6 3|7 2|7 3|6 4|6 5|7 4|8 -5|8 -4|8 -3|9 -3|8 -2|8 -1|9 -2|9 -1|8 0|8 1|9 0|9 1|8 2|8 3|9 2|8 4"}
	}		-- changing this is not supported
}

Can't run the mod with unicode strings in the tag names

Hello,

I encountered an issue where I was unable to use the script if the map have tag names which include unicode characters.

When creating the autorun.lua script, you end up with text like that :

["text"] = "\u00c0 d\u00e9truire",
["last_user"] = "ptitlaby",
["force"] = "player"

which can't be parsed by lua as far as it seems.

I found a workaround for my case, modifying the file auto.py around the step "building autorun.lua" :

Old code :

mapInfoLua = re.sub(r'"([^"]+)" *:', lambda m: '["'+m.group(1)+'"] = ', f.read().replace("[", "{").replace("]", "}"))

New code :

mapInfoLuaJSON = json.loads(f.read())
mapInfoLua = json.dumps(mapInfoLuaJSON, ensure_ascii=False)
mapInfoLua = re.sub(r'"([^"]+)" *:', lambda m: '["'+m.group(1)+'"] = ',mapInfoLua.replace("[", "{").replace("]", "}"))

I'm not sure how good this work around is

Take into account freeze_daytime when deciding what daytimes to process.

python2 runtime failure

  File "auto.py", line 24
    print("\r{}{}".format(arg, " " * (tsiz-len(arg))), end="\r\n" if tsiz <= len(arg) else "")
                                                          ^
SyntaxError: invalid syntax

This part works with python3 however.

`workthread.join()` freezing only in crop.py

workthread.join() is still freezing on occasion, unsure what the workthread is doing or why it's freezing as nothing is being logged, it is the waitKill workthread, and factorio did already get killed and it did fully die (no remnants of its process is still running).

I'm not entirely sure of the purpose of that workthread as it is spawned up and then immediately joined on.

When I canceled it after leaving it at that point for 20 minutes it printed this when I Ctrl+c'd it:

^CProcess ForkPoolWorker-55:
Process ForkPoolWorker-57:
Process ForkPoolWorker-64:
Process ForkPoolWorker-51:
Process ForkPoolWorker-66:
Process ForkPoolWorker-59:
Process ForkPoolWorker-53:
Process ForkPoolWorker-65:
Process ForkPoolWorker-56:
Process ForkPoolWorker-54:
Process ForkPoolWorker-58:
Process ForkPoolWorker-60:
Process ForkPoolWorker-61:
Process ForkPoolWorker-62:
Process ForkPoolWorker-63:
Process ForkPoolWorker-52:
sh: 1: taskkill: not found
killed factorio
disabling FactorioMaps mod
cleaning up
Traceback (most recent call last):
  File "auto.py", line 339, in <module>
    auto(*sys.argv[1:])
  File "auto.py", line 250, in auto
    crop(outFolder, otherInputs[0], otherInputs[1], otherInputs[2], basepath)
  File "/home/overminddl1/Games/Factorio/tmp/mods/L0laapk3_FactorioMaps_2.4.2/crop.py", line 66, in crop
    if progressQueue.get(True):
  File "<string>", line 2, in get
  File "/usr/lib/python3.6/multiprocessing/managers.py", line 757, in _callmethod
    kind, result = conn.recv()
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
    task = get()
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 335, in get
    res = self._reader.recv_bytes()
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
KeyboardInterrupt
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
    task = get()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 334, in get
    with self._rlock:
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
    task = get()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
    task = get()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
    task = get()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
    task = get()
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.6/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 334, in get
    with self._rlock:
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 334, in get
    with self._rlock:
  File "/usr/lib/python3.6/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
    task = get()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 334, in get
    with self._rlock:
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
    task = get()
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 334, in get
    with self._rlock:
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
    task = get()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
KeyboardInterrupt
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
    task = get()
  File "/usr/lib/python3.6/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 334, in get
    with self._rlock:
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
    task = get()
  File "/usr/lib/python3.6/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
    task = get()
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 334, in get
    with self._rlock:
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.6/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
    task = get()
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 334, in get
    with self._rlock:
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
    task = get()
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
    task = get()
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 334, in get
    with self._rlock:
  File "/usr/lib/python3.6/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 334, in get
    with self._rlock:
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 334, in get
    with self._rlock:
  File "/usr/lib/python3.6/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
  File "/usr/lib/python3.6/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
    task = get()
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 334, in get
    with self._rlock:
  File "/usr/lib/python3.6/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 334, in get
    with self._rlock:
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 334, in get
    with self._rlock:
KeyboardInterrupt
  File "/usr/lib/python3.6/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
  File "/usr/lib/python3.6/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 334, in get
    with self._rlock:
  File "/usr/lib/python3.6/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
  File "/usr/lib/python3.6/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
  File "/usr/lib/python3.6/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
KeyboardInterrupt
  File "/usr/lib/python3.6/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt

Interestingly I only hit Ctrl+c once, I'm guessing it's printing KeyboardInterrupt for each subprocess. At this point, however, it was frozen again with no further output, so after waiting a while I hit Ctrl+c again and it printed this and finally died:

^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/util.py", line 262, in _run_finalizers
    finalizer()
  File "/usr/lib/python3.6/multiprocessing/util.py", line 186, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 597, in _terminate_pool
    cls._help_stuff_finish(inqueue, task_handler, len(pool))
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 582, in _help_stuff_finish
    inqueue._rlock.acquire()
KeyboardInterrupt

A few interesting things to notice:

First, when Ctrl+c was hit it tried running taskkil /im factorio.exe, which is a Windows command (the heck?! Python should not need those.)...

It seems the subprocesses were all stuck waiting on messages, the only one different was:

  File "/home/overminddl1/Games/Factorio/tmp/mods/L0laapk3_FactorioMaps_2.4.2/crop.py", line 66, in crop
   if progressQueue.get(True):

For note mp.cpu_count() is returning 16 (the number of cores in this system).

Some testing shows that in crop.py this section:

        while len(files) > 0:
                workers = pool.map_async(partial(work, imgsize=imgsize, folder=folder, queue=progressQueue), files, 128)
                for _ in range(len(files)):
                        print(repr(files))
                        if progressQueue.get(True):
                                doneSize += 1
                                progress = float(doneSize) / originalSize
                                tsiz = tsize()[0]-15
                                print("\rcrop {:5.1f}% [{}{}]".format(round(progress * 100, 1), "=" * int(progress * tsiz), " " * (tsiz - int(progress * tsiz))), end="")
                workers.wait()
                files = [x for x in workers.get() if x]
                if len(files) > 0:
                        time.sleep(10 if len(files) > 1000 else 1)

I put a print(repr(files)) just after the for _ in range(len(files)): line and it was only called once, printing out:

['-12 -18 0 0 1\n', '-12 -17 0 0 3\n', '-11 -18 0 0 0\n', '-11 -17 0 0 1\n', '-12 -16 0 72 3\n', '-12 -15 0 0 3\n', '-11 -16 0 72 3\n', '-11 -15 0 328 2\n', '-12 -14 0 392 2\n', '-11 -14 0 392 0\n', '-10 -18 8 0 5\n', '-10 -17 8 0 f\n', '-9 -18 0 0 0\n', '-9 -17 232 0 5\n', '-10 -16 72 72 f\n', '-10 -15 72 328 f\n', '-9 -16 232 72 f\n', '-9 -15 232 328 a\n', '-10 -14 72 488 a\n', '-9 -14 0 488 0\n', '-8 -18 360 0 5\n', '-8 -17 360 0 f\n', '-7 -18 0 0 0\n', '-7 -17 264 0 5\n', '-8 -16 424 72 f\n', '-8 -15 424 232 f\n', '-7 -16 264 40 f\n', '-7 -15 264 488 b\n', '-8 -14 424 488 a\n', '-7 -14 0 488 3\n', '-7 -13 0 424 2\n', '-6 -18 424 0 5\n', '-6 -17 424 0 f\n', '-5 -18 0 0 1\n', '-5 -17 488 0 7\n', '-6 -16 456 232 f\n', '-6 -15 456 488 f\n', '-5 -16 488 232 f\n', '-5 -15 488 488 f\n', '-6 -14 456 488 b\n', '-6 -13 0 424 2\n', '-5 -14 264 488 f\n', '-5 -13 264 488 b\n', '-5 -12 0 328 2\n', '-4 -18 104 0 5\n', '-4 -17 104 0 f\n', '-3 -18 40 0 5\n', '-3 -17 328 0 f\n', '-4 -16 104 232 f\n', '-4 -15 264 488 f\n', '-3 -16 328 232 f\n', '-3 -15 328 456 f\n', '-4 -14 264 488 f\n', '-4 -13 264 488 b\n', '-3 -14 456 456 f\n', '-3 -13 456 488 e\n', '-4 -12 0 328 2\n', '-3 -12 456 328 8\n', '-2 -18 72 0 5\n', '-2 -17 296 0 f\n', '-1 -18 232 0 5\n', '-1 -17 232 0 f\n', '-2 -16 296 232 f\n', '-2 -15 296 456 f\n', '-1 -16 296 168 f\n', '-1 -15 296 360 e\n', '-2 -14 296 456 f\n', '-2 -13 296 424 e\n', '-1 -14 296 168 c\n', '-1 -13 232 424 8\n', '-2 -12 136 104 8\n', '0 -18 360 0 5\n', '0 -17 360 0 f\n', '1 -18 456 0 4\n', '1 -17 456 0 c\n', '0 -16 360 200 f\n', '0 -15 264 456 e\n', '1 -16 456 200 c\n', '1 -15 168 456 8\n', '0 -14 264 168 c\n', '0 -13 264 424 8\n', '2 -18 232 0 4\n', '2 -17 232 0 c\n', '2 -16 232 200 c\n', '2 -15 232 456 8']

Upon which it seems to permanently freeze at progressQueue.get(True).

It only seems to be happening on 'night' as well, these is the log up to that point:

GregOver2
GregOver2c
factorio path: /home/overminddl1/Games/Factorio/tmp/bin/x64/factorio
output folder: script-output/FactorioMaps/GregOver2
checking for updates
enabling FactorioMaps mod
[GAME]  138.417 Created atlas bitmap 8192x2432 [linear-minification, mipmap, terrain]                                                                                                                          libpng warning: iCCP: profile 'ICC PROFILE': 3042AF0Dh: exceeds application limits
libpng warning: iCCP: CRC error
libpng warning: iCCP: profile 'ICC PROFILE': E042E80Fh: exceeds application limits
libpng warning: iCCP: CRC error
libpng warning: iCCP: profile 'ICC PROFILE': F097D00Fh: exceeds application limits
libpng warning: iCCP: CRC error
libpng warning: iCCP: profile 'ICC PROFILE': D00DCD0Dh: exceeds application limits
libpng warning: iCCP: CRC error
[GAME]  168.751 Warning LuaGameScript.cpp:1545: luaremove_path error: Scripts can only use file space inside of script-output                                                                                  Processing GregOver2/10/nauvis/day (1 of 2)
crop 100.0% [================================================================================================================================================================================================]
ref  100.0% [================================================================================================================================================================================================]
zoom 100.0% [================================================================================================================================================================================================]
Processing GregOver2/10/nauvis/night (2 of 2)
crop   0.0% [                                                                                                                                                                                                ]['-12 -18 0 0 1\n', '-12 -17 0 0 3\n', '-11 -18 0 0 0\n', '-11 -17 0 0 1\n', '-12 -16 0 72 3\n', '-12 -15 0 0 3\n', '-11 -16 0 72 3\n', '-11 -15 0 328 2\n', '-12 -14 0 392 2\n', '-11 -14 0 392 0\n', '-10 -18 8 0 5\n', '-10 -17 8 0 f\n', '-9 -18 0 0 0\n', '-9 -17 232 0 5\n', '-10 -16 72 72 f\n', '-10 -15 72 328 f\n', '-9 -16 232 72 f\n', '-9 -15 232 328 a\n', '-10 -14 72 488 a\n', '-9 -14 0 488 0\n', '-8 -18 360 0 5\n', '-8 -17 360 0 f\n', '-7 -18 0 0 0\n', '-7 -17 264 0 5\n', '-8 -16 424 72 f\n', '-8 -15 424 232 f\n', '-7 -16 264 40 f\n', '-7 -15 264 488 b\n', '-8 -14 424 488 a\n', '-7 -14 0 488 3\n', '-7 -13 0 424 2\n', '-6 -18 424 0 5\n', '-6 -17 424 0 f\n', '-5 -18 0 0 1\n', '-5 -17 488 0 7\n', '-6 -16 456 232 f\n', '-6 -15 456 488 f\n', '-5 -16 488 232 f\n', '-5 -15 488 488 f\n', '-6 -14 456 488 b\n', '-6 -13 0 424 2\n', '-5 -14 264 488 f\n', '-5 -13 264 488 b\n', '-5 -12 0 328 2\n', '-4 -18 104 0 5\n', '-4 -17 104 0 f\n', '-3 -18 40 0 5\n', '-3 -17 328 0 f\n', '-4 -16 104 232 f\n', '-4 -15 264 488 f\n', '-3 -16 328 232 f\n', '-3 -15 328 456 f\n', '-4 -14 264 488 f\n', '-4 -13 264 488 b\n', '-3 -14 456 456 f\n', '-3 -13 456 488 e\n', '-4 -12 0 328 2\n', '-3 -12 456 328 8\n', '-2 -18 72 0 5\n', '-2 -17 296 0 f\n', '-1 -18 232 0 5\n', '-1 -17 232 0 f\n', '-2 -16 296 232 f\n', '-2 -15 296 456 f\n', '-1 -16 296 168 f\n', '-1 -15 296 360 e\n', '-2 -14 296 456 f\n', '-2 -13 296 424 e\n', '-1 -14 296 168 c\n', '-1 -13 232 424 8\n', '-2 -12 136 104 8\n', '0 -18 360 0 5\n', '0 -17 360 0 f\n', '1 -18 456 0 4\n', '1 -17 456 0 c\n', '0 -16 360 200 f\n', '0 -15 264 456 e\n', '1 -16 456 200 c\n', '1 -15 168 456 8\n', '0 -14 264 168 c\n', '0 -13 264 424 8\n', '2 -18 232 0 4\n', '2 -17 232 0 c\n', '2 -16 232 200 c\n', '2 -15 232 456 8']
killed factorio

At which point it freezes. Interesting that the file list doesn't output for the day crop...

Replacing the work function to just return False does not change anything, setting output shows that it looks like it's not even being called... interesting...

Tried calling map_async and even map in a variety of different ways and no callback executes, everything is getting locked on a semaphore and never being released.

Just to get it 'working' for now, commenting out all the multiprocess stuff in crop.py (haven't run into any issues with it in zoom.py yet for whatever reason) and just calling work directly in a loop on files and that works, only takes a second as well with about a hundred chunks (zoom is the one that takes a few seconds) and works fine.

Error while starting the mod

Hello, so I have tried to install this mod to factorio but every time I tried to open it, it give me this error.

I also tried to modify the files but I'm not a expert on lua or python so there wasn't much I could do to discover the cause of this error. I also tried to install 3 times the mod, to specify the save file path, the facorio path and I tried to start factorio with only this mod on the mods folder but nothing will work.
Any idea on how to fix this bug ?

Generate thumbnail image

Currently a random one from factoriomaps is just used

limit intermediate hard drive usage by sectioning very large maps

allow some kind of space usage limit before compression by sectioning one map as if it were multiple maps. This would take more time but allow for less disk usage while working.

Lightorio incompability

Nevermind my steam messed up.

Lua warning from factorio-side lua mod script

[GAME]  159.678 Warning LuaGameScript.cpp:1545: luaremove_path error: Scripts can only use file space inside of script-output

images

No proper confirmation on when factorio is actually done writing the screenshots to disk.

Hello, i have founded out after the corp process it stays blocked at 99.9% during a long time i was asking me why here is a print screen : https://i.imgur.com/Gv26fiw.png

scenario incompability: FactorioScenarioMultiplayerSpawn

Well to be short the mod/script fails me.
i have no clue whats wrong its a saved map on my machine.
when i trigger the generation it trows a error however does generate the map but the map is blank with a black dot.
it also shows that its missing files.
https://imgur.com/c1DXpOj
https://imgur.com/O5q4RZ5
https://imgur.com/a9oawuo

Change opacity with timelines and renderboxes

Change all paths to be relative to location of auto.py instead

Linux compatibility

Had a number of issues related to the python scripts and the final generated webpage.

Python script issues

It appears that the python script used is out of date (I am using Python 2.7.12), followed the instructions to pip install Pillow psutil and it seems the latest version of psutil does not contain a psutil.BELOW_NORMAL_PRIORITY_CLASS value, thus the calls to psutil.Process(os.getpid()).nice(psutil.BELOW_NORMAL_PRIORITY_CLASS or -10) are failing. I first had to comment all those out as I didn't really care about fixing it to the current methods as I'm just trying to get it working.
The path separators appear to have // used, which is invalid in Python programs and will only incidentally/accidentally work on some limited systems. Either a path separator of / should be used, which 'works' on all systems Python supports but is still not the recommended way, or using os.path.join(...) (which is indeed used in 'most' placed) is the proper way to join paths in all cases. 'Fixed' all those via / as there were a number of constants around...
There really should be more information about what the subprocess queues work status is as they seemed to hang on quite a number of occasions, even waiting over an hour did not see them complete, I had to force the subprocess count to just 1 to get them to work, at which case they worked properly. Did not look into what was happening as forcing the multiprocess count to 1 fixed it, perhaps a race condition or so.
Although there is a commandline option of --factorio=path/to/factorio it seems to be defaulting checks to a variety of locations, none of which would just be ../../bin/x64/factorio, which is the default executable location for the *.tar.gz download, thus forcing me to put in it's full path every time.
It appears to try killing Factorio with a taskkill command, which is not a valid program command, it should instead send an appropriate signal to the process instead, which will work everywhere that factorio runs on.
The Popen command has invalid argument types passed to it, it is not a shell style command, rather it needs to be passed a list where the first element is the program and each additional element is a command line argument (sans quotes as this is, again, not a shell expansion).
The subprocess.call calls are being used with shell expansion, but shell expansion mode is not enabled via shell=True.

A Quick and Dirty patch file to fix the above issues is:

diff --git a/auto.py b/auto.py
index f2545f7..9b9540d 100644
--- a/auto.py
+++ b/auto.py
@@ -22,16 +22,16 @@ python = sys.executable
 args = sys.argv[1:]
 kwargs = {}
 args = filter(parseArg, args)
-foldername = args[0] if len(args) > 0 else os.path.splitext(os.path.basename(max([os.path.join("..\\..\\saves", basename) for basename in os.listdir("..\\..\\saves") if basename not in { "_autosave1.zip", "_autosave2.zip", "_autosave3.zip" }], key=os.path.getmtime)))[0]
+foldername = args[0] if len(args) > 0 else os.path.splitext(os.path.basename(max([os.path.join("../../saves", basename) for basename in os.listdir("../../saves") if basename not in { "_autosave1.zip", "_autosave2.zip", "_autosave3.zip" }], key=os.path.getmtime)))[0]
 savenames = args[1:] or [ foldername ]
 
 possiblePaths = [
-    "C:\\Program Files\\Factorio\\bin\\x64\\factorio.exe",
-    "D:\\Program Files\\Factorio\\bin\\x64\\factorio.exe",
-    "C:\\Games\\Factorio\\bin\\x64\\factorio.exe",
-    "D:\\Games\\Factorio\\bin\\x64\\factorio.exe",
-    "C:\\Program Files (x86)\\Steam\\steamapps\\common\\Factorio\\bin\\x64\\factorio.exe",
-    "D:\\Program Files (x86)\\Steam\\steamapps\\common\\Factorio\\bin\\x64\\factorio.exe"
+    "C:/Program Files/Factorio/bin/x64/factorio.exe",
+    "D:/Program Files/Factorio/bin/x64/factorio.exe",
+    "C:/Games/Factorio/bin/x64/factorio.exe",
+    "D:/Games/Factorio/bin/x64/factorio.exe",
+    "C:/Program Files (x86)/Steam/steamapps/common/Factorio/bin/x64/factorio.exe",
+    "D:/Program Files (x86)/Steam/steamapps/common/Factorio/bin/x64/factorio.exe"
 ]
 try:
     factorioPath = next(x for x in ([kwargs["factorio"]] if "factorio" in kwargs else possiblePaths) if os.path.isfile(x))
@@ -40,9 +40,9 @@ except StopIteration:
 
 print(factorioPath)
 
-psutil.Process(os.getpid()).nice(psutil.ABOVE_NORMAL_PRIORITY_CLASS or 5)
+#psutil.Process(os.getpid()).nice(psutil.ABOVE_NORMAL_PRIORITY_CLASS or 5)
 
-basepath = kwargs["basepath"] if "basepath" in kwargs else "..\\..\\script-output\\FactorioMaps"
+basepath = kwargs["basepath"] if "basepath" in kwargs else "../../script-output/FactorioMaps"
 workthread = None
 
 
@@ -100,7 +100,7 @@ if os.path.isfile("autorun.lua"):
 print("enabling FactorioMaps mod")
 def changeModlist(newState):
     done = False
-    with open("..\\mod-list.json", "r") as f:
+    with open("../mod-list.json", "r") as f:
         modlist = json.load(f)
     for mod in modlist["mods"]:
         if mod["name"] == "L0laapk3_FactorioMaps":
@@ -108,7 +108,7 @@ def changeModlist(newState):
             done = True
     if not done:
         modlist["mods"].append({"name": "L0laapk3_FactorioMaps", "enabled": newState})
-    with open("..\\mod-list.json", "w") as f:
+    with open("../mod-list.json", "w") as f:
         json.dump(modlist, f, indent=2)
 
 changeModlist(True)
@@ -150,7 +150,7 @@ try:
 
 
         print("starting factorio")
-        p = subprocess.Popen(factorioPath + ' --load-game "' + savename + '" --disable-audio --no-log-rotation')
+        p = subprocess.Popen([factorioPath, '--load-game', savename, '--disable-audio', '--no-log-rotation'])
 
         if not os.path.exists(datapath):
             while not os.path.exists(datapath):
@@ -169,14 +169,14 @@ try:
 
         for screenshot in latest:
             print("Cropping %s images" % screenshot)
-            if 0 != call('%s crop.py %s %s' % (python, screenshot, basepath)): raise RuntimeError("crop failed")
+            if 0 != call('%s crop.py %s %s' % (python, screenshot, basepath), shell=True): raise RuntimeError("crop failed")
 
 
             def refZoom():
                 print("Crossreferencing %s images" % screenshot)
-                if 0 != call('%s ref.py %s %s' % (python, screenshot, basepath)): raise RuntimeError("ref failed")
+                if 0 != call('%s ref.py %s %s' % (python, screenshot, basepath), shell=True): raise RuntimeError("ref failed")
                 print("downsampling %s images" % screenshot)
-                if 0 != call('%s zoom.py %s %s' % (python, screenshot, basepath)): raise RuntimeError("zoom failed")
+                if 0 != call('%s zoom.py %s %s' % (python, screenshot, basepath), shell=True): raise RuntimeError("zoom failed")
             if screenshot != latest[-1]:
                 refZoom()
             else:
@@ -184,7 +184,7 @@ try:
                 if p.poll() is None:
                     p.kill()
                 else:
-                    os.system("taskkill /im factorio.exe")
+                    os.system("killall factorio")
                 if savename == savenames[-1]:
                     refZoom()
                 else:
@@ -230,7 +230,7 @@ except KeyboardInterrupt:
     if p.poll() is None:
         p.kill()
     else:
-        os.system("taskkill /im factorio.exe")
+        os.system("killall factorio")
     raise
 
 finally:
diff --git a/crop.py b/crop.py
index 02f9997..505c91a 100644
--- a/crop.py
+++ b/crop.py
@@ -24,10 +24,10 @@ def work(line, imgsize, folder):
 
 if __name__ == '__main__':
 
-    psutil.Process(os.getpid()).nice(psutil.BELOW_NORMAL_PRIORITY_CLASS or -10)
+    #psutil.Process(os.getpid()).nice(psutil.BELOW_NORMAL_PRIORITY_CLASS or -10)
 
-    subname = "\\".join(sys.argv[2:5])
-    toppath = os.path.join((sys.argv[5] if len(sys.argv) > 5 else "..\\..\\script-output\\FactorioMaps"), sys.argv[1])
+    subname = "/".join(sys.argv[2:5])
+    toppath = os.path.join((sys.argv[5] if len(sys.argv) > 5 else "../../script-output/FactorioMaps"), sys.argv[1])
 
     basepath = os.path.join(toppath, "Images", subname)
     
diff --git a/ref.py b/ref.py
index ed597f0..b6af396 100644
--- a/ref.py
+++ b/ref.py
@@ -72,10 +72,10 @@ def getBase64(number, isNight): #coordinate to 18 bit value (3 char base64)
 
 if __name__ == '__main__':
 
-    psutil.Process(os.getpid()).nice(psutil.BELOW_NORMAL_PRIORITY_CLASS or -10)
+    #psutil.Process(os.getpid()).nice(psutil.BELOW_NORMAL_PRIORITY_CLASS or -10)
 
 
-    toppath = os.path.join((sys.argv[5] if len(sys.argv) > 5 else "..\\..\\script-output\\FactorioMaps"), sys.argv[1])
+    toppath = os.path.join((sys.argv[5] if len(sys.argv) > 5 else "../../script-output/FactorioMaps"), sys.argv[1])
     datapath = os.path.join(toppath, "mapInfo.json")
     maxthreads = mp.cpu_count()
 
diff --git a/zoom.py b/zoom.py
index 14fdefd..fe04f29 100644
--- a/zoom.py
+++ b/zoom.py
@@ -102,10 +102,10 @@ def thread(basepath, pathList, surfaceName, daytime, start, stop, last, allChunk
 
 if __name__ == '__main__':
 
-       psutil.Process(os.getpid()).nice(psutil.BELOW_NORMAL_PRIORITY_CLASS or -10)
+       #psutil.Process(os.getpid()).nice(psutil.BELOW_NORMAL_PRIORITY_CLASS or -10)
 
 
-       toppath = os.path.join((sys.argv[5] if len(sys.argv) > 5 else "..\\..\\script-output\\FactorioMaps"), sys.argv[1])
+       toppath = os.path.join((sys.argv[5] if len(sys.argv) > 5 else "../../script-output/FactorioMaps"), sys.argv[1])
        datapath = os.path.join(toppath, "mapInfo.json")
        basepath = os.path.join(toppath, "Images")
        maxthreads = mp.cpu_count()
@@ -155,7 +155,7 @@ if __name__ == '__main__':
                                                                                        allChunks.append((pos[0]*(2**threadsplit) + i, pos[1]*(2**threadsplit) + j))
                                                                                        queue.put(queue.qsize())
 
-                                                               threads = min(len(allChunks), maxthreads)
+                                                               threads = 1 # min(len(allChunks), maxthreads)
                                                                processes = []
 
                                                                print("%s %s %s %s" % (pathList[0], str(surfaceName), daytime, pathList))
@@ -191,4 +191,4 @@ if __name__ == '__main__':
                                                                        processes.append(p)
                                                                for p in processes:
                                                                        p.join()
-
\ No newline at end of file
+

Webpage issues

The index.html has a weird <script src="/cdn-cgi/apps/head/ThhfSzyp6GI_EE-HfsckZnD6ThE.js"> tag at the top, which is a usual cloudflare tag that won't work anywhere else and is causing errors in the generated page.
Doesn't work via a file:// schema but works via http:///https://, appears to be because of some improper scripting via some third-party library.

Notes

Once worked around these issues it seems to work quite well, fantastic idea. :-)

implement multisurface link API

Give warning popup to disable the mod

Hello,

I cannot walk anymore with the current save game since I tried out this mod, Crafting and walking are not working. How can I solve this?

Already tried to remove the mod from the mod folder and disabled it in the game as well from the save game but the problems stays. ATM I am 21 IRL days in the save so I do not want lose this save game.

Separate map tags to snapshot level

Map tags could potentially have different textures when the mods are changed troughout versions.

syntax error in auto.py

File "auto.py", line 112
print(f"PID: {pid}")
^
SyntaxError: invalid syntax

Add setting for lower quality as standard definition

Script hangs with other directory

Environment: Windows 10 x64, Steam, Python 3.6.1, Factorio 0.17.50, Mod 3.4.0

With ModMyFactory, there is no mod directory, it is a symlink or junction instead, suggesting to unpack the script somewhere else, e.g. in %factoriodata%/mapper/%zipfilename%/ instead of %factoriodata%/mods/%zipfilename%/

In this case the auto.py script hangs waiting for the latest.txt to be created, even after factorio.exe is dead again (e.g. closed by user from the main menu).

I'd suggest adding a point 4.5 to the manual, telling users to wait for the game to start, hang and be killed -- and when the "Welcome to FactorioMaps, back to main menu" popup opens, something has gone wrong. Probably also to look out for a factorio console message, indicating that mapping has started -- if there is one.

Group nearby tags

Error: unrecognized option '--disable-audio'

Microsoft Windows [Version 10.0.17763.1]
(c) 2018 Microsoft Corporation. All rights reserved.

H:\Factorio\Game\0.16.51\mods\L0laapk3_FactorioMaps_1.2.1>python auto.py
C:\Program Files\Factorio\bin\x64\factorio.exe
Checking for updates
enabling FactorioMaps mod
..\..\script-output\FactorioMaps\0.16 Series
cleaning up
creating autorun.lua from autorun.template.lua
starting factorio
unrecognised option '--disable-audio'

General options:
  -h [ --help ]                      display help
  --version                          show version information
  -v [ --verbose ]                   enable verbose logging
  -c [ --config ] PATH               config file to use
  --no-log-rotation                  don't rotate log file
  --mod-directory PATH               Mod directory to use

Running options:
  -s [ --map2scenario ] arg          map to scenario conversion
  -m [ --scenario2map ] arg          scenario to map conversion
  --apply-update arg                 immediately apply update package
  --create FILE                      create a new map
  --map-gen-settings FILE            Map generation settings for use with
                                     --create. See data/map-gen-settings.exampl
                                     e.json
  --start-server FILE                start a multiplayer server
  --start-server-load-scenario FILE  start a multiplayer server and load the
                                     specified scenario
  --start-server-load-latest         start a multiplayer server and load the
                                     latest available save
  --until-tick TICK                  run a save until given map tick
  --mp-connect ADDRESS               start factorio and connect to address
  --load-game FILE                   start Factorio and load a game in
                                     singleplayer
  --benchmark FILE                   load save and run benchmark
  --benchmark-ticks N (=1,000)       number of ticks for benchmarking. Default
                                     is 1000
  --force-opengl                     use OpenGL for rendering
  --force-d3d                        use Direct3D for rendering

Server options:
  --port N                           network port to use
  --bind ADDRESS[:PORT]              IP address (and optionally port) to bind
                                     to
  --rcon-port N                      Port to use for RCON
  --rcon-password PASSWORD           Password for RCON
  --server-settings FILE             Path to file with server settings. See
                                     data/server-settings.example.json

killing factorio
ERROR: The process "factorio.exe" not found.
disabling FactorioMaps mod
reverting autorun.lua
Traceback (most recent call last):
  File "auto.py", line 155, in <module>
    time.sleep(1)
KeyboardInterrupt

H:\Factorio\Game\0.16.51\mods\L0laapk3_FactorioMaps_1.2.1>

Mod no longer works(3.5.0), Factorio(1.17.69)

After installing the mod, version 3.5.0, I can no longer load or start any game, instead I get this message.

This only happens with version 3.5.0. I tried with 3.4.0 and I could start and load just fine.

Extra info:
The FactorioMaps is the only mod installed.
Installed from the ingame modManager, but also tried from the released download on github.
Factorio version: 0.17.69(build 47194, win64, steam)