Comments (11)
@Double-y Thanks for your interest in the package! To clarify, are you saying there are existing tokenizers for Japanese that you think it would be nice to implement as an option in the way that the tokenizers package is used, or that you think it would be good to structure unnest_tokens in a more general way so that users can supply their own tokenizers?
from tidytext.
@juliasilge What if we allowed users to pass a function (or a string like we already do) to the token
argument? I could set that up!
from tidytext.
Here's one implementation (along with test cases). You can pass a custom function to token =
(and any extra arguments to ...
). Please try it out @juliasilge and @Double-y!
from tidytext.
@dgrtwo This looks really good on an initial work through on my part, and I like the way the user interacts with the function here, the flexibility, etc. Really nice! @Double-y, let us know what you think as you try it out.
from tidytext.
Thanks for considering this feature @juliasilge, @dgrtwo! Yes, I know a tokenizer for Japanese called RMeCab. I'll try it on 338cc6f
from tidytext.
Yeah, it works :)
This might be a very specific matter for Japanese but it is tokenised with part of speech detection at the same time for efficiency. But in the current unnest_tokens framework, I can't preserve the part of speech information right? The output of tokenizer is like this.
If you have a good idea to preserve this information as well. It's gonna be amazing.
from tidytext.
That's great that it works! Thanks for bringing this up; I think this improves flexibility/usability for the package.
In English, we have been doing parts of speech detection separately (after unnesting) using a join with a data set that's in the package; at this point this has been separate from the unnest_tokens function because we have been working for thinking within tidy data principles, etc. Let me/us think about that.
from tidytext.
Yeah, as for English (and probably for most alphabetical languages too), separating those steps seems to be better.
That demand will be very specific to Japanese, so I don't think it should be implemented in this package either.
I just wanted to know if there's a good way to do it. Don't worry too much about it.
Thank you!
from tidytext.
One suggestion for keeping parts of speech (I agree it unfortunately wouldn't fit in the tidytext package since it's a very specific need) is to use unnest manually. I haven't been able to get RMeCab to work on my computer so this is a rough guess I haven't tested:
RMeCabWrapper <- function(...) {
ret <- RMeCab::RMeCabC(...)
lapply(ret, function(e) dplyr::data_frame(pos = names(e), word = e))
}
d %>%
tidyr::unnest(RMeCabWrapper(text))
This (or something similar to it) should be able to create two columns: pos
(with the parts of speech) and word
(with the words).
from tidytext.
@dgrtwo Thank you for the suggestion! It looks good. I'll try it.
from tidytext.
This issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with a reprex: https://reprex.tidyverse.org) and link to this issue.
from tidytext.
Related Issues (20)
- Filter Harmful Words? HOT 3
- Parts_of_speech Expected Behavior HOT 3
- unnest_tokens on large corpus with limited RAM HOT 3
- Error in qr.lm(thetasims[, k], qx) HOT 7
- Release tidytext 0.3.4 HOT 2
- Make hunspell a Suggested dependency? HOT 2
- scales::wrap_format() does not work with scale_*_reordered HOT 4
- Change to new cran checks badge URL HOT 3
- include algo of tidy autostemmer HOT 1
- Release tidytext 0.4.0 HOT 1
- Release tidytext 0.4.1 HOT 1
- Support tidy.BTM
- [Accessibility issue] Vignettes lacks alt= in plots HOT 1
- Replacement of `token = "tweets"` HOT 3
- tidytext idf negative values due to wrong counting of number of documents HOT 2
- Error in cast_dfm() function HOT 2
- Speed improvement for bind_tf_idf HOT 5
- Disregard this issue. HOT 1
- Switch over to using native pipe
- Release tidytext 0.4.2 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tidytext.