Improve file loading by loading files from MPQ dynamically about diabloweb HOT 19 CLOSED

What's more, music was streamed directly from the MPQ, same with most of the voice lines. The Tristram theme alone is 25MB so it couldn't be kept in memory and there's some fancy streaming mechanism to support that.

from diabloweb.

I'm going to close this as I believe this is resolved now. I also replaced spawn.mpq with a smaller version in 1.0.24 that uses gzip compression and MP3 files.

from diabloweb.

Not sure what you mean, you can't dynamically read from a user supplied file, either get it all or get nothing. It's kept in JS memory and WASM code reads parts of it as needed.

What's more, if the game needs to read from a file, it is not possible to do it asynchronously since the game execution can't be paused. That means all file data has to be readily available.

from diabloweb.

Well MPQ is a container, it contains many files, not all files are needed to run the game.
The game loads files from the container dynamically as needed. (Example if you are in Town you just need the Town files loaded in order to display the level and so on). As any container or virtual filesystem, you can get the offset in the container of the file you are targeting and instead of reading the whole MPQ you can read chunks of it that contain the file. The game does this, that's why when it loads a new level it shows a progress bar, so this doesn't need to be asynchronous at all. All you need to implement some wrapper around MPQ reading that will do HTTP calls with range defined (Documentation: https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests)

Hopefully now is more clearer what I meant.

from diabloweb.

As I said, that's not possible. First of all, the large MPQ is not stored on a server but is instead uploaded by the player, so there's no HTTP requests to be made, everything is already in memory.

As for Spawn, it is not possible to do HTTP requests because the game needs the file now and I can't pause the script to wait until the file is loaded, it's just not possible in JS execution model. I'd have to rewrite half the game to make it go back to the event loop while the file loads and then resume back to where it was.

from diabloweb.

The game is already paused while it's displaying a loading screen with a progress bar when it's loading a new level correct?

So basically you are saying that a game from 1996 loaded the whole 500MB MPQ in memory because it needed all the files at an instant notice.

I don't believe computers back then had more than 32MB of RAM running Windows 95, so the game loaded from disk only the needed data.

How I see it, we just need to replace the routines that load the game from "disk" or memory with a simulation of reading from a disk, that's what HTTP range requests do, the game won't care if the data comes from disk, or from memory or from a HTTP server. Also because you are displaying a loading screen the operation of reading data doesn't matters if it is async or sync, because reading from disk was also blocking back then.

I don't know the game code too well, but I will look at it and do a pull request if I manage to do this, because for me it doesn't sounds impossible at all.

Maybe for your specific use case, drag and dropping the file you have locally is enough for you, but I would want to load data from an MPQ via a HTTP server without downloading the whole file before hand.

from diabloweb.

It's not paused, it's actively loading stuff. While the code is executing, there is no way for it to process events coming from other sources, such as user input or network. For it to happen, the code needs to issue a request, return, and then once the request is done a new function will be called with the response data. It would require a complete overhaul of the loading code.

Regular programs are very different - they can read from disk synchronously, aka the program is paused until the data is read.

If you want to make the full game MPQ available online (which is probably illegal btw), I'd start with replacing all WAV files with MP3 and using gzip compression instead of pkware, in my experience it dropped archive size from 500MB down to 220MB, which is a lot more manageable.

from diabloweb.

Well I know about the legal concerns of putting stuff online if you don't own the copyright, but there are other use cases that are not illegal. Keeping backups if you really own the game is allowed and also preventing bit rot for older software is also a noble cause.

Back to the topic at hand I don't think it's a massive rewrite because Emscripten already supports this and abstracts it away, here is an example: https://emscripten.org/docs/porting/files/Synchronous-Virtual-XHR-Backed-File-System-Usage.html

from diabloweb.

Actually you're right, there are synchronous HTTP requests available in workers, that should work. In regular JS nobody really uses that anymore since writing async code in JS is easy, but in this case it could be a thing.

If you want to do that, it should be fairly simple - all you need is modify the function get_file_contents in game.worker.js.

from diabloweb.

Glad we are on the same page 👍

from diabloweb.

You can also extract files from your MPQ and load them direct:
https://github.com/diasurgical/devilution/blob/master/Source/diablo.cpp#L268

from diabloweb.

That too, but it would flood the server with hundreds of requests.

I just tried it and I got async loading to work, I'll add it to the code but I'm not going to enable it because it's not needed in my version (I need to fully load spawn.mpq to store it in local DB so it's immediately available from there on).

Commit 61628eb includes remote loading (with a large function to make loading work in 1MB chunks and cache already loaded chunks). If you clone the repository, add diabdat.mpq to public folder and modify App.js to always start the retail version (this line), it should work. Same with shareware if you remove the code that loads it in loader.js#52.

from diabloweb.

@ZaDarkSide wrote:

So basically you are saying that a game from 1996 loaded the whole 500MB MPQ in memory because it needed all the files at an instant notice.

I don't believe computers back then had more than 32MB of RAM running Windows 95, so the game loaded from disk only the needed data.

I am player circa 1998 and I confirm we used low amounts of RAM and even crappy CD-ROM drives at the time. This made us to learn phases of game's asset data loading quite a bit.

The main phase (during which the most of level data was loaded) happened during a level loading (going through stairs or the town portal). CD drive was juggling like mad during that minute or two.

Then sometimes it occurred moments when some level assets (most likely a unique monster or some new, unseen item) were seemingly still unavailable in the OS memory. So ATM a lot smaller amount of MPQ I/O was being introduced then (now during the gameplay).

But as our CD drives (and maybe disks) were crappy and the game's algorithm was synchronous, what was supposed to be a transparent action often became a mega lag scene. Which paused the gameplay on screen (but not the background music, IIRC) until this small loading up was over. After that game usually got like 2-3x speedup for the time interval equal to that spent during the lag scene.

So yes, the original game did some extra load operations from MPQ during level. And the harder was a level, the more lags we experienced (I believe, namely due to that mechanism).

EDIT: I remember how I hated the lag which occurred very often when fighting Hell Spawns and Soul Burners – you being warrior finally become close to the succubus, click to hit it with a sword and... BAM! The game pauses, you wait, it resumes and the witch is already gone further away from you again!

from diabloweb.

Well, there is always a pagefile, and I believe the game allocated much more than 20 megs of virtual memory.

But I don't know a bit about theme playing. Maybe some files were neither compressed nor encrypted inside MPQ so loading their data wasn't so disastrous? Maybe it's time for me to read some C finally as the devilution has progressed a lot : )

from diabloweb.

Music was encrypted but not compressed. Reading sources won't get you far, since that part is inside Storm.dll.

from diabloweb.

@sskras wrote:

Then sometimes it occurred moments when some level assets (most likely a unique monster or some new, unseen item) were seemingly still unavailable in the OS memory. So ATM a lot smaller amount of MPQ I/O was being introduced then (now during the gameplay).

I think this happens when there were dialog, like everything with SFX_STREAM in https://github.com/diasurgical/devilution/blob/master/Source/effects.cpp

Maybe also dead player animation but I'm not sure of the code yet.

from diabloweb.

I added experimental support for MP3/zlib compression, it seems to work fine if I just put in my optimized diabdat.mpq in public folder. I reduced it to 231MB by encoding all wav files as mp3 and compressing with zlib instead of pkware, and removing spawn.mpq from the archive.

from diabloweb.

That's a great improvement 👍, how can I replicate your results?

from diabloweb.

Extract all files from MPQ using whatever MPQ editor. Re-encode all audio files (I used ffmpeg), keep the extension. Pack everything back.

Either use it as it is used now, or put it in public folder and change the line mentioned above in App.js to 'true'.

from diabloweb.