exponential-decay / skeleton-container-test-suite-generator Goto Github PK

This new signature has this folder structure: Metadata/Properties.plist; Index/Document.iwa
The skeleton file generated has: Index/Metadata/Properties.plist; Index/Metadata/Document.iwa

Hey Ross
I just did a rebuild of the suite for v104 PRONOM and am getting fresh issues with fmt/1190. This format was causing problems last release that I'd thought I'd fixed for siegfried by allowing the Position attribute to start from 0 rather than 1 but there seems to be another issue.

For reference, here's the signature:

This is how siegfried parses that:

Because there are two ByteSequence elements, both with a BOFoffset Reference attribute, siegfried interprets it as two sequences that are both anchored off the beginning of the file. Those min/max offsets it interprets as being relative to the start of the file.

The container generator seems to parse it differently, with the first sequence relative to the BOF and the second sequence relative to the first (so the min/max offsets for the second sequence are calculated from where that first sequence ends).

Siegfried doesn't match the skeleton file because the second sequence appears past the max offset if you calculate it from the BOF but if you calculate it from the end of the first sequence it should match.

I suspect the container generator's interpretation matches the intent of the author of this signature (because of the Position 0, Position 1 business) but on the other hand there are examples in the DROID signature file where you legitimately have multiple byte sequences anchored from the BOF so I don't want to change the way siegfried parses that as it will break those. Also, if you want to have two sequences relative to each other, you'd expect this to be two SubSequence elements within a single ByteSequence & not two ByteSequences.

Aaargh!

These signatures both have multiple patterns with offsets SubSeqMinOffset="0" SubSeqMaxOffset="1024"

The generator is incorrectly positioning sequences in the pattern. The first sequence in these patterns is put in the middle of the range (at offset 500 something). But the next sequence doesn't appear until about offset 2096 (i.e. adding the max offsets + length of the previous sequence). This is too far out (if the first sequence is at 500, should be at a max of about 1596 or something.
offsets.zip

Problem with both these files, different signature patterns each.

    <ContainerSignature Id="24100" ContainerType="OLE2">
        <Description>Corel Presentation</Description>
        <Files>
            <File>
                <Path>PerfectOffice_MAIN</Path>
                <BinarySignatures>
                    <InternalSignatureCollection>                    
                        <InternalSignature ID="24100">
                            <ByteSequence Reference="BOFoffset">
                                <SubSequence Position="1" SubSeqMinOffset="0" SubSeqMaxOffset="0">
                                    <Sequence>'ÿWPC'</Sequence>
                                </SubSequence>
                            </ByteSequence>
                            <ByteSequence Reference="BOFoffset">
                                <SubSequence Position="1" SubSeqMinOffset="8" SubSeqMaxOffset="0">
                                    <Sequence>0F 0F 02</Sequence>
                                </SubSequence>
                            </ByteSequence>
                        </InternalSignature>
                    </InternalSignatureCollection>
                </BinarySignatures>
            </File>
        </Files>
    </ContainerSignature>
    
    <ContainerSignature Id="24200" ContainerType="OLE2">
        <Description>Compound WordPerfect for Windows Document 6 onwards</Description>
        <Files>
            <File>
                <Path>PerfectOffice_MAIN</Path>
                <BinarySignatures>
                    <InternalSignatureCollection>                    
                        <InternalSignature ID="24200">
                            <ByteSequence Reference="BOFoffset">
                                <SubSequence Position="1" SubSeqMinOffset="0" SubSeqMaxOffset="0">
                                    <Sequence>'ÿWPC'</Sequence>
                                </SubSequence>
                            </ByteSequence>
                            <ByteSequence Reference="BOFoffset">
                                <SubSequence Position="1" SubSeqMinOffset="8" SubSeqMaxOffset="0">
                                    <Sequence>01 0A 02 02</Sequence>
                                </SubSequence>
                            </ByteSequence>
                        </InternalSignature>
                    </InternalSignatureCollection>
                </BinarySignatures>
            </File>
        </Files>
    </ContainerSignature>

fmt-892-container-signature-id-24200.zip

fmt-877-container-signature-id-24100.shw.zip

The three ole2 files:

• fmt-1371-container-signature-id-48000.opd
• fmt-1372-container-signature-id-48010.opd
• fmt-1373-container-signature-id-48020.opd

Have signature syntax like:

        <ContainerSignature Id="48000" ContainerType="OLE2">
            <Description>Omnipage 10 Document</Description>
            <Files>
                <File>
                    <Path>PageManager/page1</Path>
                </File>
                <File>
                    <Path>Version/Data</Path>
                    <BinarySignatures>
                        <InternalSignatureCollection>                    
                            <InternalSignature ID="48000">
                                <ByteSequence Reference="BOFoffset">
                                    <SubSequence Position="1" SubSeqMinOffset="0" SubSeqMaxOffset="0">
                                        <Sequence>04 01 00 00 65 00 00 00 43 61 65 72 65 20 4F 6D 6E 69 70 61 67 65 20 4F 50 44 20 31 30 2E 30</Sequence>
                                    </SubSequence>
                                </ByteSequence>
                            </InternalSignature>
                        </InternalSignatureCollection>
                    </BinarySignatures>
                </File>       
            </Files>
        </ContainerSignature>

        <ContainerSignature Id="48010" ContainerType="OLE2">
            <Description>Omnipage 12 Document</Description>
            <Files>
                <File>
                    <Path>PageManager/page1</Path>
                </File>
		 <File>
		    <Path>Version/Data</Path>
                    <BinarySignatures>
                        <InternalSignatureCollection>                    
	                        <InternalSignature ID="48010">
	                            <ByteSequence Reference="BOFoffset">
	                                <SubSequence Position="1" SubSeqMinOffset="0" SubSeqMaxOffset="0">
	                                    <Sequence>04 01 00 00</Sequence>
	                                </SubSequence>
	                            </ByteSequence>
	                        </InternalSignature>
	                    </InternalSignatureCollection>
                    </BinarySignatures>
                </File>              
            </Files>
        </ContainerSignature>
        
        <ContainerSignature Id="48020" ContainerType="OLE2">
            <Description>Omnipage 18 Document</Description>
            <Files>
                <File>
                    <Path>Document/Page1</Path>
                </File>
                <File>
                    <Path>Document/Data</Path>
                    <BinarySignatures>
                        <InternalSignatureCollection>                    
                            <InternalSignature ID="48020">
                                <ByteSequence Reference="BOFoffset">
                                    <SubSequence Position="1" SubSeqMinOffset="0" SubSeqMaxOffset="0">
                                        <Sequence>52 4F 53 00</Sequence>
                                    </SubSequence>
                                </ByteSequence>
                            </InternalSignature>
                        </InternalSignatureCollection>
                    </BinarySignatures>
                </File>              
            </Files>
        </ContainerSignature>

And the PageManager/page1 path is interpreted as a directory.

Compare it to SIARD though and the terminating slash can be used to tell us it actually is a directory: header/siardversion/2.1/.

So, it looks like we need another check in the code that enables us to create a placeholder file rather than just a directory.

ranges.zip

These container signatures both have ranges in the pattern e.g. 00 00 00 00 [01-04]

The range seems to be ignored by the generator (just puts 0 bytes there)

The results would be pretty awful right now, but it would be good to fix (approx. effort 3 hours) to be better able to maintain the rest of the code.

fmt-1196-container-signature-id-31020.zip

I can never get the SIARD skeleleton file to match. It is driving me mad! It looks OK ... i.e. when you decompress it "has" that darned directory.

But I don't think it really does have that directory at all...

...it just has a single empty file that has that directory as part of its filename.

BEHOLD!!! ...

Is this a PRONOM issue or a suite issue???

(both in SF and DROID)

All sequences:

23200
23210
23220
23100
23110
23120

It's not clear what the signature says as it is split into two separate BOF sequences.

This new signature starts with four 0s and then a range at the fifth offset: "\x05" - "\x0f".
The skeleton file has another zero byte at the fifth offset.

https://groups.google.com/d/msg/droid-list/v4CHVddELaM/IhmBcN0Vk_oJ

# We may have a bitmask... deal with it here...
# 
# From matt palmer:
# DROID 6 should, in fact, be capable of identifying bit-fields, 
# although there has not been a signature which uses this so far.  
# The byteseek library which DROID uses to process signatures has 
# an "all-bitmask" operator &, and an "any-bitmask" operator ~.
# For example, if you wanted to specify that bit 4 must match 
# (but you don't care about the other bits), you could write 
# [&08].  Of if you wanted to specify that a byte must be odd, 
# then you could write [&01]. Or more complex multi-bit masks as 
# well. I guess you could also test for it not matching using the 
# DROID syntax for an inverted set !: [!&01].

The container generator always exits with an error code (sys.exit(1)) following this commit

This is breaking my github action: https://github.com/richardlehane/builder/runs/5283775500?check_suite_focus=true as it halts on any step that returns an error code. There's a workaround where you can put "continue-on-error: true" to force the github action to keep running but this is a bit of a hack as would hide any real errors thrown by your script.

Jython has documented that it wants to move toward a Python 3 release, but it's not here yet.

• Jython3 MVP: https://www.jython.org/jython-3-mvp.html
• "Future of Jython": jython/jython#24

My own code is maybe a day away from being Pyhton3 compatible.

Adding unit tests to the code is maybe a few more days on top of that. One benefit of this entire utility, at least from a technical debt perspective, is that the skeleton files themselves provide an integration test of sorts. That's how we can make surface/aesthetic changes and know whether they are successful. When we get into more complex territory, e.g. signature parsing and output, it gets a little harder. When we come to add Python 3 lets see what we can add by way of unit testing.

Additional questions:

• Are there any other languages that would lend themselves well to writing this utility?
• What skew was introduced between the standard skeleton generator module and the container module?
  • Can they be combined and distributed as a sub-module?
  • What would combining repositories look like?