mostlyserious / riak-js Goto Github PK
View Code? Open in Web Editor NEWRiak client for Javascript
Home Page: http://riak-js.org
Riak client for Javascript
Home Page: http://riak-js.org
Coffee-script: 1.0.0
Node.js: 0.3.1
npm: 0.2.13
riak-js: 0.3.0beta4
$ cake dev node.js:50 throw e; // process.nextTick error, or 'error' event on first tick ^ Error: unrecognized option: --no-wrap at OptionParser.parse (/usr/local/lib/node/.npm/coffee-script/1.0.0/package/lib/optparse.js:34:17) at /usr/local/lib/node/.npm/coffee-script/1.0.0/package/lib/command.js:234:29 at Object.run (/usr/local/lib/node/.npm/coffee-script/1.0.0/package/lib/command.js:25:5) at Object. (/usr/local/lib/node/.npm/coffee-script/1.0.0/package/bin/coffee:7:27) at Module._compile (node.js:329:23) at Object..js (node.js:337:12) at Module.load (node.js:260:25) at loadModule (node.js:232:12) at require (node.js:272:14) at Object. (/usr/local/bin/coffee:11:18)
db.get('test', undefined, {keys: "stream"})(sys.puts, function(e,m,r) { sys.puts('ERROR:' + r) })
ERROR:Couldn't parse it as JSON
This shouldn't happen (don't attempt to convert to JSON if it's streaming)
Everything is buffered now.
Streaming data back to the client would be ideal for:
Hey,
My goal was to override encode
and decode
in order to allow special processing for specific buckets.
A real world example would be something like:
decode = (data) ->
json = JSON.parse data
json.id = @key
json
encode = (data) ->
delete data.id
JSON.stringify data
Unfortunately I currently can not provide custom Meta objects currently as they are passed to the Meta constructor anyway. What I do at the moment looks like:
Meta = require 'riak-js/lib/meta'
Meta::encode = (data) ->
super unless @bucket == 'foo'
# do something
A nice way to do that would be:
class FooMeta extends Meta
decode = (data) ->
# ...
meta = new FooMeta()
db.save 'foo', {hello: 'world'}, meta, (err, res) ->
# hi
What do you think? :)
I may be using this wrong, but it worked before version 0.3.0beta6
db.getAll('users', { where: { email: req.body.user.email} }, function(err, users, meta) {});
This would return just the user objects with the specified email. Now it returns all users regardless of the email input.
I can't get riak-js to connect to Riak. I'm running a cluster and trying to connect to 192.168.1.103 but it seems that riak-js is always trying to connect somewhere else. I can use Riak's webinterface so I know the instance is up. What's the correct way to connect different IPs? I tried many different variations of getClient({host:'192.168.1.103'}) but it just doesn't seem to work.
Make tests picking one client randomly out of multiple talking to different interfaces.
It'll reveal areas where the APIs need to be more even, or flaws in meta
(which should be completely independent).
Is there any support for that comming? I see a ranges branch in repo is it Key filtering?
According to Riaks documentation it is possible to get a random key if using POST against a bucket without specifying the key.
I simulate this in riak-js like this:
client = require('riak-js').getClient();
client.save('test', 0, 'this is a test');
which performs a POST, but how can I get the "Location" in order to get the random generated key?
Ideally I want to build a wrapper around riak-js that will take care of:
The latter could also be achieved by using something like HAProxy but to start with we could support something super simple.
Should it belong to riak-js? Probably not.
Setup test environment to check speed and correctness.
I really don't know what would be ideal.
But as a default there's probably something better than id-per-session (think i saw that in Basho's javascript client) or id-per-riak-client (what we currently have, riak-js
).
What about grabbing the host name? We could end up with something like riak-js-Francisco-Treacys-MacBook-Pro.local
So I had this idea of pulling hostname
while setting up the client, but it'd be more convenient to do that synchronously. Not sure.
Make it transparent to the user to install this dependency (which itself depends on protobuf). Hook it into an npm install script - or whatever they call it.
Get that in asap!
When retrieving data using get()
, the dictionary passed to getClient()
is included in the returned meta-data.
var riak = require('riak-js'), sys = require('sys');
db = riak.getClient( { debug: false, someKey: 'someValue' });
db.save('foo', 'bar', {}, function(err, data) {
db.get('foo', 'bar', function(err, data, meta) {
console.log(sys.inspect(meta, false, 99));
db.remove('foo', 'bar');
});
});
Returns:
{ usermeta: {},
debug: false,
api: 'http',
clientId: 'riak-js',
binary: false,
raw: 'riak',
someKey: 'someValue', // <-- Should this be here?
contentEncoding: 'utf8',
links: [],
host: 'localhost',
accept: 'multipart/mixed, application/json;q=0.7, */*;q=0.5',
responseEncoding: 'utf8',
bucket: 'foo',
key: 'bar',
contentType: 'application/json',
vclock: 'a85hYGBgzGDKBVIsTOunKmUwJTLmsTJMZX1yjC8LAA==',
lastMod: 'Thu, 03 Mar 2011 12:29:09 GMT',
etag: '"4RQh4xZBXMpfQQQvhVTO8K"',
contentRange: undefined,
acceptRanges: undefined,
statusCode: 200 }
I don't know if this is by design, but I stumbled across it when I accidentally gave the constructor some unrelated data, that suddenly caused riak-js not to accept my modified header data. I have unfortunately not been able to reproduce the error reliably...
I have semi-randomly tried this on a few of the beta's, as well on @latests, all with the same result.
How can I change the default timeout (60000) in maprep? According mapper's source, run() seems to be accepting [options,callback] but it doesn't seem to work at least with {timeout:999999} option. I tried to add timeout in bunch of different methods related to mapred that according source should accept options but can't get it to work :/
If I'm not mistaken, a major benefit of Luwak is dealing with very large files. Rather than having to hold these large files in memory, why not allow for the data to be streamed through the API. This would be pretty killer IMO.
Same api as normal, the library would just have to handle the stream internally. This can be done using util.pump();
var readstream = fs.createReadStream("/path/to/large/file");
db.saveLarge(<hash>, stream, function(err) {
console.log('stream finished.');
});
Using the older versions of npm to install the riak-js module works fine, but using the latest version doesn't seem to install the lib directory:
mjr@test1-01:$ npm install riak-js$ node
[email protected] ./node_modules/riak-js
mjr@test1-01:
require("riak-js");
Error: Cannot find module './lib/index'
at Function._resolveFilename (module.js:320:11)
at Function._load (module.js:266:25)
at require (module.js:348:19)
at Object. (/home/mjr/node_modules/riak-js/index.js:1:80)
at Module._compile (module.js:404:26)
at Object..js (module.js:410:10)
at Module.load (module.js:336:31)
at Function._load (module.js:297:12)
at require (module.js:348:19)
at [object Context]:1:1
mjr@test1-01:~$ ls node_modules/riak-js
Cakefile docs erl_src index.js LICENSE package.json README.md spec src
The key generated by Riak during an POST request is not returned in the metadata.
I'm getting this error on the latest node-proxy. Is this normal?
(node) warning: possible EventEmitter memory leak detected. 11 listeners added. Use emitter.setMaxListeners() to increase limit.
at Pool. (events.js:101:17)
at Object.proxyRequest (/usr/local/lib/node/.npm/http-proxy/0.3.1/package/lib/node-http-proxy.js:185:7)
at /opt/node/bb/lib/vhost.js:256:31
at /opt/node/bb/virt-dispatch.js:11:5
at Server. (/opt/node/bb/virt-dispatch.js:20:56)
at Server.emit (events.js:45:17)
at HTTPParser.onIncoming (http.js:862:12)
at HTTPParser.onHeadersComplete (http.js:85:31)
at Socket.ondata (http.js:787:22)
at Socket._onReadable (net.js:623:27)
I tried importing fairly large amounts of data into riak using Node.js as a middle-man, my program continuously uses more and more memory. Cutting away the cruft, I could cut it down to this: https://gist.github.com/756279 (the setTimeout should probably be dropped in preference for just launching a new round of save()'s in the callback, when in_flight==0).
When executing the program on my machine, i get a memory usage like so (X-axis is seconds.):
Before it dies of excessive memory use.
I'm using the latest beta (4, I believe) + Node.js HEAD.
removeLink({bucket: 'users', key: '[email protected]'}) does not work, cause the links are stored in a URL-escaped HTTP header
e.g. removeLink({bucket: 'users', key: 'foo%40bar.com'}) works fine
Since npm 1.0 ignores files that are excluded by a .gitignore if no .npmignore is present, your lib
dir is getting stripped out of the package.
There are a few ways to fix this:
"files" : ["lib"]
to the package.json. (This is like explicitly un-ignoring lib.)"scripts": { "preinstall" : "cake install" }
and make coffee-script a regular dependency instead of a devDependency.Also, I notice that you have some vows tests. It'd be good to add a "test" script in there as well, so that you'll be set to take advantage of the auto-testing coming in the next few months.
Is there a way to send authentication through the getClient() request? I know we can specify host and port, but what if we have http authentication running somewhere in the middle?
something like:
var db = require('riak-js').getClient({auth: 'key:secret', host: 'xx.xx.xx.xxx', port: '8080'});
or even
var db = require('riak-js').getClient({host: 'key:[email protected]', port: '8080'});
Installed forever 0.4.1 (which depends on winston, which depends on riak-js) using npm 1.0rc and on first run it gives such error:
% forever -v
node.js:134
throw e; // process.nextTick error, or 'error' event on first tick
^
Error: Cannot find module './lib/index'
at Function._resolveFilename (module.js:320:11)
at Function._load (module.js:266:25)
at require (module.js:348:19)
at Object. (/Users/jakub/node_modules/forever/node_modules/winston/node_modules/riak-js/index.js:1:80)
at Module._compile (module.js:404:26)
at Object..js (module.js:410:10)
at Module.load (module.js:336:31)
at Function._load (module.js:297:12)
at require (module.js:348:19)
at Object. (/Users/jakub/node_modules/forever/node_modules/winston/lib/winston/transports/riak.js:10:14)
Is it a bug in riak-js?
At the moment, getAll
provides a plain object with some properties, not a real Meta
.
It should behave like a get
does with an HTTP 300 Multiple Choices
, so those Meta
can be reused in subsequent operations.
At the moment there is some testing but I feel it should be easier. Evaluate using vows or expresso (but needs to play nicely with coffeescript).
We should have amazing coverage just as Ripple.
On with the "airplane" theme, we could generate fake passengers with http://github.com/Marak/faker.js
Also make use of Sean's riak_kv_test_backend?
http://github.com/seancribbs/ripple/blob/ed9a65ee4c830236ec6c61808a2c1cba54dc7945/riak-client/erl_src/riak_kv_test_backend.erl
In general, stdout is where "informational" level output goes, like a w3c compatible log stream, for example. Debug logging should write to stderr.
Your code in Client.log currently just calls console.log. It should call console.error at minimum (ideally, it would be nice if I could pass you a stream for logging).
I can fork and send you a pull request if that's easier.
it seems that spaces in the id are not escaped:
db.save('airlines', 'ARG 1', { name: 'Aerolíneas Argentinas 1' , fleet: 1, european: false }
PUT /riak/airlines/ARG 1
db.save('airlines', 'ARG 2', { name: 'Aerolíneas Argentinas 2' , fleet: 2, european: false }
PUT /riak/airlines/ARG 2
db.count('airlines')
1
When using riak-js with nginx you can pop up with problem that there's no Host http request header is provided for http request, so nginx won't proxy it. Can you put a default Host header with value of host param given to getClient function?
Hey, Frank, i faced a strange thing today.
I was doing something like:
client.head("bucket", "missingkey", {}, function(err, data, meta){
if( err ){
// it goes here if meta.statusCode == 404
}
});
it seems a bit wrong to me, because i didn't expect it as error, i was just checking for key existance and in my case it was a normal response. So to check if it's really 404 i gotta:
if( err ){
if( err.statusCode == undefined ){
// handle some error like riak is unreachable
}
else if( err.statusCode == 404 ){
// woot it's not found
}
else {
// something else
}
}
Shouldn't 404 be treated like normal response?
output from airport-test.js when riak is not available:
macpro1:test siculars$ node airport-test.js
[riak-js] PUT /riak/test-airports/EZE
[riak-js] PUT /riak/test-airports/BCN
[riak-js] PUT /riak/test-airports/AMS
[riak-js] PUT /riak/test-airports/CDG
[riak-js] PUT /riak/test-airports/MUC
[riak-js] PUT /riak/test-airports/JFK
[riak-js] PUT /riak/test-airports/HKK
[riak-js] PUT /riak/test-airports/MEX
[riak-js] PUT /riak/test-flights/KLM-8098
[riak-js] PUT /riak/test-flights/AFR-394
[riak-js] PUT /riak/test-flights/CPA-112
[riak-js] PUT /riak/test-flights/IBE-5624
[riak-js] PUT /riak/test-flights/ARG-714
[riak-js] PUT /riak/test-flights/DLH-4001
[riak-js] PUT /riak/test-flights/AMX-1344
[riak-js] PUT /riak/test-flights/AMX-1346
[riak-js] PUT /riak/test-flights/KLM-1196
[riak-js] PUT /riak/test-flights/CPA-729
[riak-js] PUT /riak/test-flights/ARG-909
[riak-js] PUT /riak/test-flights/IBE-4418
[riak-js] PUT /riak/test-airlines/KLM
[riak-js] PUT /riak/test-airlines/AFR
[riak-js] PUT /riak/test-airlines/AMX
[riak-js] PUT /riak/test-airlines/ARG
[riak-js] PUT /riak/test-airlines/DLH
[riak-js] PUT /riak/test-airlines/IBE
[riak-js] PUT /riak/test-airlines/CPA
[riak-js] POST /mapred/
[riak-js] POST /mapred/
[riak-js] GET /riak/test-airlines/
[riak-js] GET /riak/test-airports/
[riak-js] GET /riak/test-flights/
[riak-js] ERROR: Couldn't reach localhost:8098
[riak-js] ERROR: Couldn't reach localhost:8098
pastie for readability: http://pastie.org/897596
Hi. I'm trying to update a object. Works the first time but not the second..
Create it..
> db.save("pers", "1", { title: "hans" })
PUT /riak/pers/1
false
> db.getAll("pers")
POST /mapred
false
> [ { meta: { bucket: 'pers', key: '1', vclock: 'a85hYGBgzGDKBVIsTO+1QzKYEhnzWBmuW5Qf48sCAA==' }, data: { title: 'hans' } } ]
Update it the first time, which works.
> db.save("pers", "1", { title: "peter" })
PUT /riak/pers/1
false
> db.getAll("pers")
POST /mapred
false
> [ { meta: { bucket: 'pers', key: '1', vclock: 'a85hYGBgymDKBVIsTO+1QzKYEhnzWBmuW5Qf44MIszUnsezm+QOVeAKSyAIA' }, data: { title: 'peter' } } ]
but if I try to update it again
> db.save("pers", "1", { title: "andreas" })
PUT /riak/pers/1
false
it doesnt work:
> db.getAll("pers")
POST /mapred
false
> [ { meta: { bucket: 'pers', key: '1', vclock: 'a85hYGBgymDKBVIsTO+1QzKYEhnzWBmuW5Qf44MIszUnsezm+QOVeAKSyAIA' }, data: { title: 'peter' } } ]
I'm not sure if its a bug or if I just got something wrong.. Any ideas?
The Riak protocol supports the auto-generation of keys on insert if one isn't provided (simply posting to /riak/bucketname). I could be wrong but after looking at the docs/source, I found no way to do this in riak-js. Seems like it would be pretty trivial to add support for this. If I get any time to learn coffeescript I could take a look at implementing it I suppose.
It doesn't seem to happen all the time though. Need to dig into it (check how other HTTP clients deal with this; would it make sense to replace the client?)
Also, emit 'error' event.
I dunno if it's critical, since riak is no point of failure DB, but, in case of when all instances are dead, I couldn't get any callback on command at all. I use this code:
var db = Riak.getClient(riakConfig);
db.save(type, key, data, options)(function(data, meta){
sys.puts(1);
});
and it never hits sys.puts(1); I see following in console output:
[riak-js] PUT /riak/xxx/ppp
{"message":"ECONNREFUSED, Connection refused","stack":"Error: ECONNREFUSED, Connection refused\n at IOWatcher.callback (net:854:22)\n at node.js:772:9","errno":111,"syscall":"connect"}
Error: ECONNREFUSED, Connection refused
That's it. Maybe I should catch errors like that somewhere else?
An ODM or whatever you wanna name it. Just like mongoose, visionmedia/rapid or Ripple.
I have already implemented something super basic with coffeescript classes, dunno how well it plays with js.
Anyway this functionality should belong to a project outside riak-js.
I've ported the Meta object to coffee script, using the master branch of coffee script. Jeremy just merged in a big patch, so the syntax is a bit different if you already know coffeescript. Also, once I get further in, I'll be using the coffeescript watch command to generate the JS files in /lib from /src.
I wanted to move the link encoding/decoding code to Meta and add some tests for that. The PBC client won't use it, but it won't hurt to have it there.
Any thoughts on the coffee script branch so far? http://github.com/technoweenie/riak-js/compare/coffee
It seems like the callbacks in riak-js return after the HTTP request has been sent, not when the response is received.
This means race conditions are created.
Check out this sample code gist for an example:
Hi Frank,
I am running node.js 0.2.4, riak 0.13.0, riak-js 0.3.0beta2 via npm. Following code taken from http://riakjs.org/:
db.save('airlines', 'KLM', {fleet: 111, country: 'NL'}, { links:
[{ bucket: 'flights', key: 'KLM-8098', tag: 'cargo' },
{ bucket: 'flights', key: 'KLM-1196', tag: 'passenger' }]
});
db.walk('airlines', 'KLM', [["_", "flight"]]);
fails with:
TypeError: Object [object Object] has no method 'link'
at [object Object].walk (/usr/local/lib/node/.npm/riak-js/0.3.0beta2/package/lib/http_client.js:112:15)
at Object.<anonymous> (/home/factons/1.js:12:4)
at Module._compile (node.js:462:23)
at Module._loadScriptSync (node.js:469:10)
at Module.loadSync (node.js:338:12)
at Object.runMain (node.js:522:24)
at Array.<anonymous> (node.js:756:12)
at EventEmitter._tickCallback (node.js:55:22)
at node.js:772:9
Thanks,
Ozgur
Guys, do you think that this agent.maxSockets can make a queue instead of doing requests parallel in HTTP client of riak-js? Maybe make some setting to set it higher, because default value is just 5.
I have a bucket called session with ~3000 keys like these:
"keys":["styjSNlsbwEIYCfgNINc9nku.CqdstcngbnJhTAns5M+m18Yau72bwzcc3EsGEI5vv10","G3RSuq4f1SFrVBK5i2eUyhOi.yOidtB6Vx7NQzIVDhpCwgoGrNPqWMbnykp473GhGdE","UJjx7wF28zSqqld36iQnLUFL.POmMSODsoBaJ5l7J9OhrnqvhtZoDgScgwyGtWzMNiE","HAqC3usgAMWBiGQkY21dttVW.rysityOevs+ENIlLuJqSZ6sF9sgZGUXBEOwba4M8rJw" ... ]
I have a function that clears all the keys in that bucket, it worked fine for other small buckets, with different (shorter and simple) key names:
clearAllSessions = function clearAllSessions(){
util.debug("[*] clearing all sessions from backend");
function cb(err, buffer, meta){
util.debug("[] clearing sessions");
if (err) throw err;
buffer.keys.forEach(function(key){
db.remove('session',key);
});
util.debug("[] cleared sessions");
}
db.get('session','?keys=true',cb);
//db.keys('session', cb);
}
The callback never gets called, and there is no error. I'm using http to talk to riak.
I've tried to make this call with keys=stream, but I get the error:
Error: Cannot convert response into application/json: Unexpected token { -- Response:
Is there an option like noJSON for get method?
May it be a problem in converting the response to JSON, or is the request being aborted?
cheers,
Daniel
[edit] I'm using node 0.4.2 and riak-js clone repo 0.3.6
I'm getting an error in riak.js
headers is undefined
[line: 269] this.type = headers['content-type'];
Dug into the riak-jquery.js file and the helper method inside of .executeInternal
// helper
function getMeta(xhr) {
var meta = new Meta(), headers = xhr.getAllResponseHeaders();
meta.headers = {};
meta.statusCode = xhr.status;
if (headers.length) {
headers.split('\r\n').forEach(function(header) {
// header of type: Content-Length: 146
var i = header.indexOf(':');
meta.headers[header.substring(0, i).toLowerCase()] = header.substring(i+2);
});
}
var location = meta.headers['location'];
meta.key = location ? location.substring(location.lastIndexOf('/')+1) : options.key;
return meta;
}
the
var meta = new Meta() need parameters passed to it otherwise it throws and error and the javascript dies.
I am running an example code which is identical to the one on the webpage:
db.get('flights', 'KLM-5034', function(err, flight, meta) {
if (err) throw err
flight.status = 'delayed'
meta.links.push({ bucket: 'airlines', key: 'IBE', tag: 'operated_by' })
db.save('flights', 'KLM-5034', flight, meta)
})
But I got almost all the time the following error ( very seldom it works without error ):
/node_modules/riak-js/lib/utils.js:84
target[k] = d.value; ^
undefined
Any clues what is happening here?
thanks!.
walk seems to work only if the returned objects are application/json. I'm using the data from the link walking tutorial, which is text/plain.
db.walk('people', 'sean', [["_", "friend"]])
POST /mapred
> { stack: [Getter/Setter],
arguments: undefined,
type: undefined,
message: 'HTTP error 500: {"lineno":477,"message":"JSON.parse","source":"unknown"}',
statusCode: 500,
notFound: false }
My guess is that Riak.mapValuesJson barfs on the non-JSON content (text/plain). I'm surprised this POSTs to /mapred rather than GETting /riak/people/sean/_,friend,1
POST /mapred HTTP/1.1
Accept: multipart/mixed, application/json;q=0.7, */*;q=0.5
Host: localhost
Connection: close
Content-Type: application/json
Content-Length: 269
{"inputs":[["people","sean"]],"query":[{"link":{"bucket":"_","tag":"friend","keep":true,"language":"javascript"}},{"reduce":{"language":"erlang","module":"riak_kv_mapreduce","function":"reduce_set_union"}},{"map":{"name":"Riak.mapValuesJson","language":"javascript"}}]}
Including generated javascript adds a lot of noise when viewing/diffing commits.
Would it be smart to remove the lib
folder altogether and add a compilation step when people install via npm? For those who clone the repo or download tarballs a simple coffee -wc --no-wrap -o lib src/*.coffee
would do the trick.
NPM has support for listing a package's development dependencies, so they can be taken care of with npm install --dev
.
diff --git a/package.json b/package.json
index c6dda55..a484413 100644
--- a/package.json
+++ b/package.json
@@ -41,5 +41,9 @@
},
"os": [ "linux", "darwin" ],
"dependencies": {},
+ "devDependencies": {
+ "vows": "*",
+ "coffee-script": "*"
+ },
"engines": { "node": ">=0.1.97" }
-}
\ No newline at end of file
+}
The actual versions may need some editing for correctness.
I know it sounds crazy and if one of my users reported this to me, I'd probably dismiss it as user error - and it may yet be a mistake I'm making and not realizing...
I have a test app (it's actually a hugely cut-down chunk of a much larger app) which lets users PUT a file onto a server. Earlier in the call stack, a riak.get is executed (in the full app, that is part of the authentication process). When the riak.get is avoided, the file upload works like a charm. When the riak.get is executed, the file upload stalls, but I have no clue why.
You can see the test program at https://gist.github.com/957127 (I didn't know if it would be couth to post a block that size in an issue...)
This issue is, unfortunately, blocking me only a day from a very important deadline, so I'd really appreciate whatever help you can offer.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.