Comments (9)
Modifying an existing archive is not really easy task.
From a file operations perspective, unless the file size (of the modified file inside the archive) stays the same, you can't just write a part of the file, you need to essentially write the whole file again. Which means you need to either read the whole thing into memory first, or write into a new file and then do an atomic move.
From a xoreos code perspective, we have no concept at all of modifying archives. We can read archives and we can write (some) archives, and those are two completely different paths. (And especially if you're talking MOD files, we don't yet do all ERF features eithers).
From a tools perspective, this should probably be a feature of the erf (and rim) command, not a separate command to modify archives. We might also want to make the erf/rim commands behave a bit like the usual rar/zip/7z comnands from an invocation stand-point, like unerf mimics unrar/unzip a bit.
I'm not sure what you mean "get a file from the archive in XML format". If you're talking getting an XML representation of a GFF, that's not something that an archive command should concern itself.
from xoreos-tools.
Sure I agree that modifying files in-place isn't feasible, so what I really mean is that it creates a new file in-place, i.e. not requiring a -o flag to be set (which is useful for batch editing if, say, one wanted to edit a script that's in multiple archives).
To me, it makes more sense to have a single "archive tool" that has --erf, --rim, --mod, ... flags, but you seem to disagree or be going for a different model and it's not super important.
As you say, the main things that would need to be implemented would be adding/removing files from an archive once it's loaded into memory - replacing could then be done by a combination of remove and add, and reading/writing is already implemented.
If there was no support for reading binary from stdin, allowing the output to be xml rather than binary would be useful (as otherwise converting to xml would require writing and reading a tmp file, which seems a shame to have to do when using two tools from the same library). But with support for reading binary from stdin (and therefore piping) this is a non-issue, and in that case I agree with you 100%.
from xoreos-tools.
what I really mean is that it creates a new file in-place, i.e. not requiring a -o flag to be set
I have no idea what you mean by that. Can you give me an example usage?
going for a different model
Yeah, I'm more going in with a Unix "do one thing and do it well" type of model, which is why I also don't think an archive tool should concern itself being able to convert GFF to XML. Though whether that "one thing" is manipulating archives or manipulating one archive format is debateable.
from xoreos-tools.
Oh, also:
As you say, the main things that would need to be implemented would be adding/removing files from an archive once it's loaded into memory
The thing is, we don't load an archive completely into memory at the moment. We load the resource index, but the individual files inside the archive are read from the file as needed.
from xoreos-tools.
Re the in-place operation, as an example, say I have a script which is included in three .rim files for the game I'm working on, and I want to update that script in all three modules at once. I could run "unrim--replace --gff file.gff --name "script.ncs" --inplace -i *" and it would automatically replace the script in all modules that have it (rather than having to use a loop to set -i and -o to the same filename). This isn't a big deal by any means; just an idea.
I imagine that it wouldn't be too difficult to modify the current code so it can load an entire archive into memory at once, rather than picking individual files out (I'd be willing to have a go implementing this if you want).
from xoreos-tools.
unrim --replace --gff file.gff --name "script.ncs" --inplace -i *
What would the --gff parameter do there?
Also, "-i *" doesn't really work like that. On Unixoid systems, the shell expands the glob, so this would expand into
unrim --replace --gff file.gff --name "script.ncs" --inplace -i foo.rim bar.rim foobar.rim
or potentially even
unrim --replace --gff file.gff --name "script.ncs" --inplace -i foo.rim bar.rim foobar.rim foo.wav 1.erf
I.e. the file names don't logically "bind" to the -i parameter. But that's just a minor thing, need to think about how to best map that use case onto parameters.
But that is an in-place replace, I don't understand what you mean with "in-place but not in-place".
I imagine that it wouldn't be too difficult to modify the current code so it can load an entire archive into memory at once
Well, you can just read the whole input stream into another MemoryReadStream
that's held in memory, yeah. But doing that always isn't all that great. We need to think about how exactly we're going there.
Doing it with an atomic move is probably better anyway. That way, you're not destroying the input while when something goes wrong, because you're holding out the swapping until everything is done.
from xoreos-tools.
The --gff parameter would indicate the location of the GFF file we want to insert into the archive. If both the -gff and -i flags are not given (meaning that both the GFF file and the archive are meant to be read from stdin)... I'm not sure how it would be best to handle that. It would be easiest to just require that one of these two inputs be a file, but this would clash with #59.
Perhaps in-place isn't the best word for it - I just mean that whatever the input filename is, that's also the output filename (so we're not creating a new file, just overwriting an existing one).
As you suggest, it would be best to write the stream to memory and then atomically write it to disk, rather than editing the existing file as it goes along and potentially breaking it if an error is reached. This could still be done in chunks, but realistically the files for Aurora-based games are small enough that I think loading the whole file into memory shouldn't be an issue (keeping in mind that the original games targeted e.g. the original Xbox, with 64MB of RAM).
from xoreos-tools.
Err, no, you can't atomically write a file from memory to disk. You write into a new file, flush the caches and then atomically move to new file over the old file, replacing it.
And the archives are big-ish, the games also read only the files within as they are needed. And especially in the later ganes, like The Witcher and the Dragon Ages, the archives are close to the 2GB limit.
from xoreos-tools.
Oh yeah of course, I was going to suggest that you write to a tmp file and change the name, but figured maybe you'd know a more elegant way to do it.
Interesting that the archives can get that big (my experience is mainly with KOTOR/KOTOR2 so I'm not so familiar with the later games). In any case, I don't know that there's really an alternative than to just read the whole file, edit it, and then write the result.
from xoreos-tools.
Related Issues (20)
- xml2gff not included in xoreos-tools-0.0.5 release? HOT 7
- XML2GFF: Tool needs to be game/encoding aware HOT 1
- XML2GFF: Serialize color codes HOT 7
- NCSDIS: Control flow analysis failure in do-loop nested in while-loop HOT 15
- NCSDIS: Analyse control flow for recursive functions, and functions with incompatible fork merging HOT 1
- FEATURE: Compile xoreos-tools as a DLL/library HOT 6
- FEATURE: Make more tools read stdin for input HOT 5
- ERF: MOD files have incorrect prefix HOT 4
- ERF: Recursion leads to infinite loop HOT 1
- Clarify XML format for xml2gff HOT 1
- GFF2XML -> XML2GFF (invalid base64 length) HOT 5
- UNOBB: Add support for KotOR2 Android obb archives HOT 29
- Issue running configure to build project: Syntax error near unexpected token `FT2 HOT 3
- xml2gff: NWN:EE invalid file format HOT 12
- error while compiling HOT 6
- another error while compiling HOT 1
- FEATURE: Add tool to convert TGA back to TPC
- CONVERT2DA: Code and documentation mismatch for the parameters HOT 5
- xml2tlk gives error when trying convert back cp1251 xml HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from xoreos-tools.