query selection on vector quantities, i.e., magnitudes about nbodykit HOT 12 CLOSED

So here is where we start to see that issue. The alternative is to give the mini-language indexing

mag[:, 0]

But the missing part is the mapping between 0 1 2 3 .. and urgiz (which is an knowlege in the painter/ reader)

so mag_u mag_r may actually make more sense . What about using mag.u mag.r?

from nbodykit.

I like the mag.u mag.r syntax. I think that makes the most sense for now

from nbodykit.

I am looking in this again. What if we want to do a selection based pos[:, 0]?

from nbodykit.

Hmm, I don't think I have a good solution for this. For the plain text painters, the positions are named columns "x", "y", "z", etc, so you could do it that way, but that only works for those specific cases.

Maybe we could setup something like an "aliases" keyword that would allow you to define "x" as "eval(pos[:,0])" or something along those lines

from nbodykit.

I don't like aliases. It adds another layer of mental complexity. What about just supporting pos[:, 0] type of selections in selection language?

from nbodykit.

@rainwoodman This has come back up in the context of slicing DataSources by Position to create sub-boxes -- did we ever make any progress on this?

from nbodykit.

Nope.
In npyquery I ended up using 'pos[0] pos[1] pos[2]' to represent pos[:, 0], pos[:, 1] pos[:, 2].
The parser in selectionlanguage needs to be extended to support indexing? Do you want me to look at this?

from nbodykit.

I think it should be relatively easy to put in indexing -- I'll take a stab at it.

I think the trickier part is if we can integrate query selection with the new stream / caching interface better. For example, one problem currently is that there's no way to even slice 'Position' for the Pandas datasource, since it is selecting based off the columns read from the file and not any columns created afterwards, i.e., 'Position'.

One potential way forward:
-- selection done based only off of the column names returned by a DataSource
-- have all DataSource classes take the select keyword, similar to cosmo and comm
-- selecting would be done before any caching, so only the relevant data is cached
-- for datasources that explicitly read named columns from files, add those columns to the data returned by readall, so that they are available for slicing

from nbodykit.

If we take "have all DataSource classes take the select keyword, similar to
cosmo and comm" approach, then it smells like a 'mixin' pattern may be of
relevance. But I am not sufficiently familiar with it.

The datasource shall know the selection in order to perform the selection
the most efficient way.

On Wed, Jun 29, 2016 at 8:59 PM, Nick Hand [email protected] wrote:

I think it should be relatively easy to put in indexing -- I'll take a
stab at it.

I think the trickier part is if we can integrate query selection with the
new stream / caching interface better. For example, one problem currently
is that there's no way to even slice 'Position' for the Pandas datasource,
since it is selecting based off the columns read from the file and not any
columns created afterwards, i.e., 'Position'.

One potential way forward:
-- selection done based only off of the column names returned by a
DataSource
-- have all DataSource classes take the select keyword, similar to cosmo
and comm
-- selecting would be done before any caching, so only the relevant data
is cached
-- for datasources that explicitly read named columns from files, add
those columns to the data returned by readall, so that they are available
for slicing

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#35 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAIbTHpwxOrN7TFSLt1_jKl68YLdwNopks5qQz8ZgaJpZM4FcLOd
.

from nbodykit.

What do you mean by your second comment? You're thinking the user would implement the selection explicitly in the readall or parallel_read function?

from nbodykit.

Also, which of these syntaxes do you like more:

1. "Position[:,0] < XX"
2. "Position[0] < XX"

where in the second syntax we assume the indexing is referring to the second dimension

from nbodykit.

closed via #203

from nbodykit.