Comments (7)
Hi Dominique,
I agree, I largely punted on the database by referring to the design docs; in retrospect, that's unfair.
@jbecla , could you add a paragraph on the database, along the lines that Dominique is proposing? The challenge: I'd need it by ~tomorrow night (need to submit by 5_am_ on Friday).
I also recall you had a Supercomputing paper on qserv -- what's the reference for it?
from adass-2015-paper.
We already cite the SC paper. I added that a while back.
from adass-2015-paper.
Ah, missed that, thanks!
from adass-2015-paper.
I'll look into that tomorrow
from adass-2015-paper.
@jbecla Actually, having re-read the whole paper, I think the best thing to to would be to just add a note that Qserv has been tested in data challenges, just like the rest of the stack. The reason for that is that we don't actually dwell much on design details or performance promises for other components either.
How does this look: https://github.com/mjuric/adass-2015-paper/blob/qserv/O3-1.pdf . In the last paragraph on page 7, I added:
Advanced prototypes of the distributed, shared-nothing, database being written for LSST – Qserv – have been tested on a 150-node cluster using 55 billion rows and 30 terabytes of simulated data (Wang et al. 2011).
I took these from Daniel's paper -- what would be the updated numbers?
from adass-2015-paper.
Great, I got distracted earlier and so I am glad to hear I don't need to write it :). The number you are quoting are capturing well the scale we tested with. Independently this summer we tested with relatively comparable data set (32TB, 5B objects, 35B sources, 172B forcedSources), the most prominent thing this time around was that we got Qserv to run reliably for days under heavy load (100+ concurrent queries consisting of a mix of easy and hard ones) and with 100x lower latency than the 2011 tests.
from adass-2015-paper.
Thanks, Jacek -- I also added a link to S15 tests page on confluence.
from adass-2015-paper.
Related Issues (7)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from adass-2015-paper.