devinacker / omgifol Goto Github PK

because omg.util.zstrip() is actually kind of broken.

It should possibly be replaced with a function that takes either a str or bytes and returns a string, so that all operations involving lump names are guaranteed to use actual strings and not bytes

Hi,

in Heretic and Hexen, full-screen grahpics like e.g. "TITLEPIC" are stored in a "raw" format instead of the usual patch format. That is, the lumps consist of 320x200 bytes, each holding a palette index.

Please allow to access them using the wad.graphics[] API.

probably allow opening a pk3/pke/etc. the same as a WAD, use LumpGroups for directories, etc.

This was already done in 301ca31 but got reverted somehow during the python 3 cleanup/merge

I'm not sure fredrik's original reasoning for creating the struct system for omgifol, but it might help improve development to replace them with custom classes instead.

One thing to look at before attempting this is how fast either of them are. Benchmarking will be needed to see if there are any significant speed differences between the two.

However if it is seen as an unnecessary change then that's reasonable too. The issues I have with the struct system are somewhat minor anyway.

Currently calling WAD.from_file() on a read-only WAD file fails:

  File "/usr/lib/python2.7/site-packages/omg/wad.py", line 256, in from_file
    w = WadIO(source)
  File "/usr/lib/python2.7/site-packages/omg/wadio.py", line 75, in __init__
    self.open(openfrom)
  File "/usr/lib/python2.7/site-packages/omg/wadio.py", line 88, in open
    self.basefile = open(filename, 'r+b')
IOError: [Errno 13] Permission denied: '/usr/share/doom/freedoom2.wad'

As can be seen at the end of the callstack the file is opened 'r+b', which is read and write.

This is unfortunate in my case since I'm only interested in examining/reading a WAD file. To work around it I copy the WAD to a temporary directory, open that, and then delete it on exit.

I keep my WADs read-only so they are not accidentally modified. Perhaps other people do too, or they should.

Also, WAD.to_file() makes me think there is no need for need for WAD.from_file() to open the file read/write.

When attempting to save a map to the wad, it would error out with

  File "mirror.py", line 40, in <module>
    if __name__ == "__main__": main(argv[1:])
  File "mirror.py", line 38, in main
    outwad.to_file(args[1])
  File "/usr/lib/python3.6/site-packages/omg/wad.py", line 275, in to_file
    self.__dict__[group].save_wadio(w)
  File "/usr/lib/python3.6/site-packages/omg/wad.py", line 138, in save_wadio
    wadio.insert(t, hs[t].data)
AttributeError: 'list' object has no attribute 'data'

This can be avoided by building nodes and adding the derivative lumps

▶ lswad trivial.wad
  name  	  size  	 offset 
MAP01   	0       	12
THINGS  	10      	12
LINEDEFS	56      	22
SIDEDEFS	120     	78
VERTEXES	16      	198
SECTORS 	26      	214
WADCSRC 	113     	240
▶ cat bug1.py
import sys
from omg import *
from omg.mapedit import *
inwad= WAD()
inwad.from_file(sys.argv[1])
ed = MapEditor(inwad.maps['MAP01'])
▶ python bug1.py trivial.wad
Traceback (most recent call last):
  File "bug1.py", line 6, in <module>
    ed = MapEditor(inwad.maps['MAP01'])
  File "/home/jon/git/doom/omgifol/omg/mapedit.py", line 180, in __init__
    self.from_lumps(from_lumps)
  File "/home/jon/git/doom/omgifol/omg/mapedit.py", line 234, in from_lumps
    raise ValueError("map is missing %s lump" % e)
ValueError: map is missing 'SECTORS' lump

trivial.zip

Hi,

papers 'RL2: Fast Reinforcement Learning via Slow Reinforcement Learning' and 'A Simple Neural Attentive Meta-Learner' use 1000 randomly generated maps with different initial and target locations. Could you give me light on how to do this?

Thanks very much

also vanilla compatible when possible (https://www.doomworld.com/vb/doom-editing/71482-importing-tall-sprites-to-work-in-vanilla/)

Has this been considered yet? It would help with keeping it up to date for users, meaning avoiding issues like Linguica's, using old versions like fredrikj's latest update.

@devinacker

I see why this #27 happens. The acs script is not written into the new wad file. The data is stored in map_editor.scripts.data.

I have tried using the demo/mirror.py to verify this. And it may be the reason why the impassable walls can be passed through.

Could you fix this? Thanks in advance.

https://github.com/GitExl/WhackEd4

Whacked4 is written in python, it might be worth seeing if we can pull features from it to add dehacked support to omgifol

Been discussing this with @devinacker, currently you can just read it like any other zip and read into the Lump classes, but there's potentially a way for omgifol to read the structure and just use LumpGroups for directories.

WadSmoosh uses omgifol for all its heavy lifting, and I've noticed that the map WADs I extract (from the source IWADs) using it don't end up with the same "map MD5 sums" that GZDoom et al expect for detecting and applying the map fixes specified in its internal data. I'd much rather let GZDoom apply its fixes than add lots of custom fix code to WadSmoosh.

According to its code, each entry in GZDoom's compatibility.txt is an MD5 sum of the map's "header, THINGS, LINEDEFS, SIDEDEFS, SECTORS, and BEHAVIOR lumps". When I try "mapchecksum e3m4" for the omgifol-extracted version versus the doom.wad version, I do indeed get a different result.

The WadSmoosh code for extracting a map from an IWAD looks like this:

def extract_map(in_wad, map_name, out_filename):
    out_wad = omg.WAD()
    ed = omg.MapEditor(in_wad.maps[map_name])
    out_wad.maps[map_name] = ed.to_lumps()
    out_wad.to_file(out_filename)

So it seems like something MapEditor.to_lumps() does introduces a change from the input data.
Bug, or feature?

Hi devinacker, when I generate the wad file, how can I set the initialized location and the goal location?

Could you offer some clues? Thank you very much.

I replaced my old omgifol scripts with these and under Python 2.4.4 they no longer worked properly. "from omg import *" threw up errors.

If it now requires Python 2.7 please update the docs to say so.

To create a PLAYPAL lump, the process is:

pal.make_bytes()
lump = omg.Lump(pal.bytes)

To create a COLORMAP lump, the process is:

lump = col.to_lump()

The API should be consistent when dealing with different types of data. I think the to_lump() function should be available on the Palette class since that is consistent with most of the rest of the API (to_file, to_Image etc)

Just for an example, I was using this with v1.4 of CHEX3.WAD, the same one that can be verified with the checksums here: https://doomwiki.org/wiki/CHEX3.WAD

This wad just so happens to be missing a P_END entry, so everything afterwards gets put into the patches group, this includes flats and map lumps that come after all of the patches in this wad.

https://github.com/sirjuddington/SLADE/blob/master/src/Utility/Polygon2D.cpp#L95-L127
https://github.com/jminor/omgifol/blob/master/demo/wad2obj.py

Being able to convert a sector to convex polygons is an incredibly useful feature. This can be used for many operations based around converting a map into other formats, or for creating 3d visualisations, increasing the applications of omgifol. Above linked are two possible sources to look at for implementation.

This code:

import omg
w = omg.WAD()
w.from_file('doom.wad')
img = w.sprites['HEADA1'].to_Image()

produces this error:

Traceback (most recent call last):
  File "image_error.py", line 6, in <module>
    img = w.sprites['HEADA1'].to_Image()
  File "omg/lump.py", line 169, in to_Image
    im.fromstring(self.to_raw())
  File "omg/lump.py", line 151, in to_raw
    pointers = unpack('%il'%width, data[8 : 8 + width*4])
struct.error: unpack requires a string argument of length 504

Happens with every graphics lump I try in a wad.

Running omgifol from Arch Linux, Python 2.7.11 with PIL version 1.1.7 (Pillow version 3.2.0)

This code:

import omg
inwad = omg.WAD()
inwad.from_file('doom.wad')
ed = omg.MapEditor(inwad.maps['E4M7'])

gives the following error:

Traceback (most recent call last):
  File "e4m7_error.py", line 6, in <module>
    ed = omg.MapEditor(inwad.maps['E4M7'])
  File "omg/mapedit.py", line 161, in __init__
    self.from_lumps(from_lumps)
  File "omg/mapedit.py", line 183, in from_lumps
    self.sidedefs = self._unpack_lump(Sidedef,   m["SIDEDEFS"].data)
  File "omg/mapedit.py", line 176, in _unpack_lump
    return [class_(bytes=data[i:i+s]) for i in range(0,len(data),s)]
  File "omg/mapedit.py", line 176, in <listcomp>
    return [class_(bytes=data[i:i+s]) for i in range(0,len(data),s)]
  File "<struct>", line 12, in __init__
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc0 in position 4: ordinal not in range(128)

Running omgifol from Arch Linux, Python 3.5.1.

Give the user the option to export a texture, built from patches, as an Image(). Currently omgifol users have to manually traverse the data and build patches themselves, but this should be an in-built feature of omgifol.

My current implementation (including transparency fix in issue #17) is as follows:

wad = omg.WAD(path)
tex = omg.txdef.Textures(wad.txdefs)

# build the textures out of the patches
for texturedef in tex:
  new_img = Image.new("RGBA", (tex[texturedef].width, tex[texturedef].height))
  for p in tex[texturedef].patches:
    pimg = wad.patches[p.name.upper()].to_Image()
    pimg = pimg.convert("RGBA")
    # try and fix transparency issues
    pixdata = pimg.load()

    width, height = pimg.size
    for y in xrange(height):
      for x in xrange(width):
        if pixdata[x, y] == (255, 0, 255, 255):
          pixdata[x, y] = (255, 255, 255, 0)

    new_img.paste(pimg, (p.x, p.y), pimg)

but this should be as simple as follows:

wad = omg.WAD(path)
tex = omg.txdef.Textures(wad.txdefs)

# get the textures
for texturedef in tex:
  new_img = tex[texturedef].to_Image()

https://github.com/devinacker/omgifol/blob/master/lump.py#L165

The format for the image used ("P") does not support transparency, and any transparent pixels in a graphic are rendered at RGB(255,0,255). This means not only will all exported sprites/patches contain magenta areas, when building textures from patches, every patch with a transparent area will put magenta pixels over the texture.

Example: https://puu.sh/uUJAa/3c86248001.png

A (computationally slow) workaround is possible currently with the following code:

patch_img = wad.patches[patch_name].to_Image()
patch_img = patch_img.convert("RGBA") # Convert to a format that supports alpha transparency

pixdata = patch_img.load()
width, height = patch_img.size
for y in xrange(height):
  for x in xrange(width):
    if pixdata[x, y] == (255, 0, 255, 255): # Find any magenta pixels
      pixdata[x, y] = (255, 255, 255, 0) # Replace with transparent pixel

This adds a significant amount of time to loading patches, especially when working with entire wads of textures such as the IWADs.