Comments (2)
Comments from Wes:
one relevant section is in pandas/core/groupby.py. here's the code
with a few comments. as you can see I haven't completely solved the
problem, there's still a 64-bit overflow possibility that I'm punting
on (this doesn't occur that much in practice except with very large
number of group keys)
def _get_compressed_labels(self):
# These are the "factorized" integer numbers for each group key
all_labels = [ping.labels for ping in self.groupings]
if self._overflow_possible:
# We're punting and doing an object hash if we're going
# to overflow a 64-bit integer. Lame, I know
# Convert the label arrays to Python tuples
tups = lib.fast_zip(all_labels)
# Pass the tuples through a hash table to compute integer labels
labs, uniques = algos.factorize(tups)
# Sort them if so desired
if self.sort:
uniques, labs = _reorder_by_uniques(uniques, labs)
return labs, uniques
else:
if len(all_labels) > 1:
# This is the "cartesian product number"
group_index = get_group_index(all_labels, self.shape)
# Pass the cartesian product number through the hash
# table again to remove sparsity
comp_ids, obs_group_ids = _compress_group_index(group_index)
else:
# Just use the integer labels from the only group key
ping = self.groupings[0]
comp_ids = ping.labels
obs_group_ids = np.arange(len(ping.group_index))
self.compressed = False
self._filter_empty_groups = False
return comp_ids, obs_group_ids
My recommendation: use a fast hash table impl, either std::hashmap or
https://github.com/attractivechaos/klib (I don't think the one used in
factor(...) is very good, and there's some wasted performance there),
to convert basic R data types into integers. Write the above helper
functions, and then you should be in business.
I haven't looked at the data.table implementation for GPL reasons but
there might be some interesting algorithmic details in there to learn
from perhaps-- though when your data is indexed and therefore
presorted this all becomes much easier
from dplyr.
Moved to dplyrRcpp
from dplyr.
Related Issues (20)
- group_split(): add a '.groups_to_names=T' type argument HOT 2
- `group_split()` not shown on the reference page HOT 1
- Modify how arrange sorts strings HOT 3
- Should `desc()` use `vec_rank()` internally on character vectors?
- Problem Group_by
- `dplyr_reconstruct` can create data.table with corrupted secondary index HOT 1
- Split `join_cols()`
- Error in the function 'across()' HOT 3
- Documentation error for count HOT 4
- `if_else` `missing` argument does not work when using `%in%` to evaluate. HOT 3
- Add a Ranking Function to Compute Tail Means HOT 1
- across(), but over two sets of vector
- filter may crash RStudio Session when combining rbind and rowwise HOT 3
- Look at dplyrover HOT 1
- `if_any` on zero-column selection yields `TRUE` HOT 2
- Typo in cumulative function docs
- `bind_rows` does not capture `tapply` output names with `.id` parameter HOT 1
- A subtle scoping flaw in pull()
- Feature Request: add `.groups = "drop"` option to `mutate` HOT 3
- Feature request: global option for summarise .groups setting HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dplyr.