Comments (14)
already included:
ä -> ae, ö -> oe, ü -> ue, ß -> ss, @ -> at, ` -> "", ' -> ""
from snakecase.
review this issue also regarding encoding stuff, after this topic is included into the new advanced r version
from snakecase.
basics about encoding:
https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/
-ascii: nr 32-127 space = 32, A = 65, etc. -> 7 bits (2^7 = 128) (most were using 8 bit bytes)
so one bit was free for any usage
<32 "unprintable" -> controlcharacters (i.e. 7 = "beep", 12 = "print",...)
-> everything good for english alphabet.
-1 byte = 8 bits = possibilities: 0-255 -> 128-255 still free
- ibm oem with some european accented characters
- many other oem character sets evolved
-> so encoding displayed on different displays turned to different character outputs
ANSI standard -> 0-127 have the same meaning for all encodings regarding this standard
= 128 non-uniform ("code pages", different nr for diff countries for example)
- problems in asia -> DBCS "double byte character set" (some in 1, some in 2)(hard to ... -> wind: AnsiNext AnsiPrev)
- problem only if change computer or language
-> Unicode!!!
- effort to single character set including all! reasonable writing sytems
unicode != 16-bit code (16 bit = 65.536)
before: someting like this: A -> 0100 0001, ...
unicode: A -> code point (theoretical concept?) completely other story how this is represented in mem or disk
A (for example) is platonic ideal. A is different to B or a, but the same as italic or bold A or any font.
unicode cosortium: figured out, which characters are really different (ß not ss and also similar looks are not same characters...)
Codepoints: U+0639 (for example; U+ for unicode and then a number) - there is no limit do space of unicode characters definitely not two bites (there are alr. more than 65.536)
- Hello in Unicode U+0048 U+0065 U+006C U+006C U+006F
Endocings (to store unicode)
-
different ways appeared to store unicode. problem was with ofthen zeroes, which often unnecessarily
doubled the amount of memory. So many people didn't use it.
-> UTF-8: 0-127 werden in 1 byte, darüber in 2,3,4,5 bzw. 6 byte gespeichert.
-> für english: utf8 = ascii
Hello in unicode: U+0048 U+0065 U+006C U+006C U+006F, in utf8 UND ascii UND ansi UND all oem ch sets:
48 65 6C 6C 6F -
3 possibilities: ucs-2 (two bytes + find out if hidh/low endian), utf-16 (16 bits), UTF-8 (well with old ascii)
-
others: utf-7 (better utf8?), utf-4 (4byte utf, but too memory expensive)
-
translation from unicode to others is ok for english alphabets, but results in ? or � if not available.
-
popular englisch-text encodings: windows-1252, iso-8859-1 aka latin-1 (both for west europe, not russian,...)
-
utf7,8,16, 32 can store every codepoint
-
need ALWAYS to know encoding of strings. plain text is not ASCII. TJER AIN'T NO SUCH THING AS PLAIN TEXT
so say if it is in utf-8, ascii, latin-1, windows 1252, ... more than 100 endocings, all different above 127 -
std metadata for encoding:
- email: Content-Type: text/plain; charset=”UTF-8″
- web servers: metatag:
-
when these info is missing, different ways are gone. For example micr. int. exp. guesses.
-
however, you can also guess encodings yourself and try a bit.
from snakecase.
see also here:
https://stat.ethz.ch/R-manual/R-devel/library/base/html/Encoding.html
Maybe it is best to set the encoding at the beginning via
Encoding(x) <- "UTF-8"
which will result in "UTF-8" (for words with special characters) and "unknown" for words without special characters (ASCII 0-127).
from snakecase.
or maybe better enc2utf8
or iconv
from snakecase.
added enc2utf8
as first step into the parsing. Currently only into unstable development and experimental
edit: resolved this, since stringr (based on stringi) seem to always encode 2 utf8 anyway
from snakecase.
propaply replace_special_characters
should take a character string of the form
c("countryset1", "countryset2", ...)
instead of a logical.
The countrysets should be handled in order of their supplyment to replace_special_characters
.
The replacement should result in only 0-127 Ascii letters, so that there can't be any collisions of between several special character replacements.
from snakecase.
sth like mentioned here https://unix.stackexchange.com/questions/171832/converting-a-utf-8-file-to-ascii-best-effort
by radovan garbik will do it...
from snakecase.
need to check out this
http://stackoverflow.com/questions/17517319/r-replacing-foreign-characters-in-a-string
in detail. especiall stringi::stri_trans_general
from snakecase.
stringi::stri_trans_general(, "Latin-ASCII")
might be the best one can get...have to look up latin
from snakecase.
or maybe just wrap the stri_trans_general function and supply the id as an argument to to any case
stri_trans_general("gro\u00df", "latin-ascii")
stri_trans_general("stringi", "latin-greek")
stri_trans_general("stringi", "latin-cyrillic")
stri_trans_general("stringi", "upper") # see stri_trans_toupper
stri_trans_general("\u0104", "nfd; lower") # compound id; see stri_trans_nfd
stri_trans_general("tato nie wraca ranki wieczory", "pl-pl_FONIPA")
from snakecase.
have to look for a language and symbol agnostic transcription for unicode...icu doesnt seem to work for everything (Latin-ASCII has some problems and ü to ue is not possible in any icu conversion). maximally it could be added as an extrafeature...
from snakecase.
Use stringi::stri_trans_general()
, when next update of stringi is available on cran
from snakecase.
implemented now, update, when I find a nice source for additional transliteration dictionaries.
Currently implemented (besides stri_trans_general()): "german"
from snakecase.
Related Issues (20)
- Make parsing_option 0 work with sep_in
- Description of title case in readme needs to be corrected
- Use fixed regexes when matching abbreviations
- Bugs in the abbreviations implementation
- add new stack overflow example as a test
- Make `abbreviations` argument work with case = `"none"`
- Change order of names and element in transliterations
- Readme image doesnt contain protect step
- Strange behaviour when abbreviating special characters
- Can't escape regex special character in transliterations HOT 2
- Set up auto-build pkgdown site with every commit to master
- return character(0) for input string = NULL
- abbreviations argument appears to be ignored with case="snake" HOT 2
- snakecase::to_any_case() causes warnings on 1st run due to UTF-8 characters HOT 1
- Change travis to github actions
- Leaving Middle Initials Capitalized when using title case HOT 2
- snake case with first letter uppercase HOT 2
- Release snakecase 0.11.1
- Test suite errors HOT 2
- Feature request: Support "kebab-case" HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from snakecase.