directories-like-keys and illegal chars problem about node-persist HOT 12 CLOSED

ok - well, true, but since we are writing the filename as keys to disk as they are (for quick retrieval) there isn't a quick solution for illegal filenames characters.

I could sanitize the keys on the way in, using something like sanitize-filename, which will strip all illegal characters out, but what you do with collisions? say you have "derp/herp" now becomes "derpherp" but then you have another key that is just "derpherp"? the latter will overwrite the first one.

Well, what if we use the replacement option, and replace each illegal character by some type of placeholder, say "derp/herp" becomes "derp__SLASH__herp", great, but then again, what I really have a key that is really "derp__SLASH__herp"? back to collision.

If you think that one of these solution is OK, then I'll do it, or If you can think of another solution, that takes in the order of hours (<20) to implement, I'm happy to work with you on that as well.

but since we are writing the filename as keys to disk as they are

we could start changing that, but then we start writing a full blown database service, which is not the point of the this project.

from node-persist.

Why not create folders when paths are specified? derp/herp would create a folder derp and place herp inside of it. For example:

/** Writes the file (and any needed directories). */
function writeFileP(file, contents, cb) {
  mkdirp(path.dirname(file), function (err, made) {
    if (err) return cb(err);
    fs.writeFile(file, contents, cb);
  });
}

from node-persist.

I could do that i guess.. , but that doesn't solve the illegal characters issue

from node-persist.

Maybe slugify the file names? https://github.com/dodo/node-slug

EDIT: I see, that still won't fix the collisions issue. What about using sanitize-filename and using a hash of the original filename then appending a small portion of that it to the end? I guess there isn't a really great solution for collisions...

from node-persist.

that last commit should resolve the directories-like-keys, i.e. Users/herp/derp/gerp/nerp/burp

As for the illegal chars..

using a hash of the original filename

Good idea

then appending a small portion

we would need the whole hash to minimize collision as much as possible, i guess i can do that

from node-persist.

the problem with hashing is that it is 1-way, which means if the service goes down and needs to reload, it won't have a way to figure out what were the original keys, without changing the data structure, and setting some sort of a metadata on each value; that, along with changing the filenames, are NOT backward compatible.... let alone the hashing performance hit.

The good news is that with hashing, we no longer need to recursively create directories even for values with directories-like-keys.

i.e.

// say key/value are
var value = "hai";
var key = "/herp/derp/nersp/+-(8`\\1231 \\\/ 123/123 #!@#!$$$$%"; // bunch of illegal chars i hope

// generated filename
var filename = crypto.createHash("md5").update(key).digest("hex"); // ced387cc2f77d9e3f5b795ed2da73a3e

file storage/ced387cc2f77d9e3f5b795ed2da73a3e should have some content like

{
"key": "/herp/derp \\1231 \\\/ 123/123 #!@#!$$$$%",
"value": "hai"
}

I guess it's doable, but that means a breaking change, which also means I get to break other things

from node-persist.

this project is slowly becoming a full blown database

from node-persist.

@generalconsensus [[email protected]] solves your directories-like-keys problem from your original post, illegal chars are yet to be solved, i'll keep this discussion open

from node-persist.

5c09328

from node-persist.

fyi- [email protected] is published, using md5 hash for filenames now.
https://github.com/simonlast/node-persist#200-change-logs
0eaf17a

from node-persist.

It seems the error persists. In my case, the file path contains "\8", which probably causes this error:

Error: ENOENT: no such file or directory, open 'C:\monitoring\storage\8d777f385d3dfec8815d20f7496026dc'
at Error (native)

Indeed, in javascript Strings, \8 and \9 are forbidden.

from node-persist.

@psuzzi I am unable to reproduce this issue. Looking at your error, looks like you're on a windows machine, so I tested this on a Windows 7, and made sure one of the hashed keys starts with an 8 (simply by using a key like "s1")

Mocha test

This is running the normal mocha test, you can see on the right one of the created keys also starts with 8 - i had to comment out the removal of the temp directories, you can see it in the screenshot at line 377 of tests/index.js just so I can those directories after the test is ran.

Manual test case

here I just created a simple temp.js file with the content of

temp.js

var nodePersist = require('node-persist');
nodePersist.initSync();
// using the key "s1" here will generate a file with the name `8ddf878039b70767c4a5bcf4f0c4f65e`
nodePersist.setItemSync('s1', '\n\n\n HELLO WORLD \n');
var s1 = nodePersist.getItemSync('s1');
console.log(s1);

can you try this out for me?, create the file above and run it

node temp.js

My test looks fine.

from node-persist.