Git Product home page Git Product logo

uni's Introduction

uni queries the Unicode database from the commandline. It supports Unicode 15.1 (September 2023) and has good support for emojis.

There are four commands: identify codepoints in a string, search for codepoints, print codepoints by class, block, or range, and emoji to find emojis.

There are binaries on the releases page, and packages for a number of platforms. You can also run it in your browser.

Compile from source with:

$ git clone https://github.com/arp242/uni
$ cd uni
$ go build

which will give you a uni binary.

README index:

Integrations

  • dmenu, rofi, and fzf script at dmenu-uni. See the top of the script for some options you may want to frob with.

  • For a Vim command see uni.vim; just copy/paste it in your vimrc.

Usage

Note: the alignment is slightly off for some entries due to the way GitHub renders wide characters; in terminals it should be aligned correctly.

Identify

Identify characters in a string, as a kind of a unicode-aware hexdump:

$ uni identify €
     CPoint  Dec    UTF8        HTML       Name (Cat)
'€'  U+20AC  8364   e2 82 ac    €     EURO SIGN (Currency_Symbol)

i is a shortcut for identify:

$ uni i h€ý
     CPoint  Dec    UTF8        HTML       Name (Cat)
'h'  U+0068  104    68          h     LATIN SMALL LETTER H (Lowercase_Letter)
'€'  U+20AC  8364   e2 82 ac    €     EURO SIGN (Currency_Symbol)
'ý'  U+00FD  253    c3 bd       ý   LATIN SMALL LETTER Y WITH ACUTE (Lowercase_Letter)

It reads from stdin:

$ head -c2 README.markdown | uni i
     cpoint  dec    utf-8       html       name (cat)
'['  U+005B  91     5b          [     LEFT SQUARE BRACKET (Open_Punctuation)
'!'  U+0021  33     21          !     EXCLAMATION MARK (Other_Punctuation)

You can use -compact (or -c) to suppress the header, and -format (or -f) to control the output format, for example you may want to generate a codepoint to X11 keysym mapping:

$ uni i -c -f '0x%(hex): "%(keysym)", // %(name)' h€ý
0x68: "h", // LATIN SMALL LETTER H
0x20ac: "EuroSign", // EURO SIGN
0xfd: "yacute", // LATIN SMALL LETTER Y WITH ACUTE

See uni help for more details on the -format flag; this flag can also be added to other commands.

Search

Search description:

$ uni search euro
     CPoint  Dec    UTF8        HTML       Name (Cat)
'₠'  U+20A0  8352   e2 82 a0    ₠   EURO-CURRENCY SIGN (Currency_Symbol)
'€'  U+20AC  8364   e2 82 ac    €     EURO SIGN (Currency_Symbol)
'𐡷'  U+10877 67703  f0 90 a1 b7 𐡷  PALMYRENE LEFT-POINTING FLEURON (Other_Symbol)
'𐡸'  U+10878 67704  f0 90 a1 b8 𐡸  PALMYRENE RIGHT-POINTING FLEURON (Other_Symbol)
'𐫱'  U+10AF1 68337  f0 90 ab b1 𐫱  MANICHAEAN PUNCTUATION FLEURON (Other_Punctuation)
'🌍' U+1F30D 127757 f0 9f 8c 8d 🌍  EARTH GLOBE EUROPE-AFRICA (Other_Symbol)
'🏤' U+1F3E4 127972 f0 9f 8f a4 🏤  EUROPEAN POST OFFICE (Other_Symbol)
'🏰' U+1F3F0 127984 f0 9f 8f b0 🏰  EUROPEAN CASTLE (Other_Symbol)
'💶' U+1F4B6 128182 f0 9f 92 b6 💶  BANKNOTE WITH EURO SIGN (Other_Symbol)

The s command is a shortcut for search. Multiple words are matched individually:

$ uni s globe earth
     CPoint  Dec    UTF8        HTML       Name (Cat)
'🌍' U+1F30D 127757 f0 9f 8c 8d 🌍  EARTH GLOBE EUROPE-AFRICA (Other_Symbol)
'🌎' U+1F30E 127758 f0 9f 8c 8e 🌎  EARTH GLOBE AMERICAS (Other_Symbol)
'🌏' U+1F30F 127759 f0 9f 8c 8f 🌏  EARTH GLOBE ASIA-AUSTRALIA (Other_Symbol)

Use shell quoting for more literal matches:

$ uni s rightwards black arrow
     CPoint  Dec    UTF8        HTML       Name (Cat)
'➡'  U+27A1  10145  e2 9e a1    ➡   BLACK RIGHTWARDS ARROW (Other_Symbol)
'➤'  U+27A4  10148  e2 9e a4    ➤   BLACK RIGHTWARDS ARROWHEAD (Other_Symbol)
…

$ uni s 'rightwards black arrow'
     CPoint  Dec    UTF8        HTML       Name (Cat)
'⮕'  U+2B95  11157  e2 ae 95    ⮕   RIGHTWARDS BLACK ARROW (Other_Symbol)

Add -or or -o to combine the search terms with "OR" instead of "AND":

$ uni s -o globe milky
     CPoint  Dec    UTF8        HTML       Name (Cat)
'🌌' U+1F30C 127756 f0 9f 8c 8c 🌌  MILKY WAY (Other_Symbol)
'🌍' U+1F30D 127757 f0 9f 8c 8d 🌍  EARTH GLOBE EUROPE-AFRICA (Other_Symbol)
'🌎' U+1F30E 127758 f0 9f 8c 8e 🌎  EARTH GLOBE AMERICAS (Other_Symbol)
'🌏' U+1F30F 127759 f0 9f 8c 8f 🌏  EARTH GLOBE ASIA-AUSTRALIA (Other_Symbol)
'🌐' U+1F310 127760 f0 9f 8c 90 🌐  GLOBE WITH MERIDIANS (Other_Symbol)

Print

Print specific codepoints or groups of codepoints:

$ uni print U+2042
     CPoint  Dec    UTF8        HTML       Name (Cat)
'⁂'  U+2042  8258   e2 81 82    ⁂   ASTERISM (Other_Punctuation)

Print a custom range; U+2042, U2042, and 2042 are all identical:

$ uni print 2042..2044
     CPoint  Dec    UTF8        HTML       Name (Cat)
'⁂'  U+2042  8258   e2 81 82    ⁂   ASTERISM (Other_Punctuation)
'⁃'  U+2043  8259   e2 81 83    ⁃   HYPHEN BULLET (Other_Punctuation)
'⁄'  U+2044  8260   e2 81 84    ⁄    FRACTION SLASH (Math_Symbol)

You can also use hex, octal, and binary numbers: 0x2024, 0o20102, or 0b10000001000010.

General category:

$ uni p Po
Showing category Po (Other_Punctuation)
     CPoint  Dec    UTF8        HTML       Name (Cat)
'!'  U+0021  33     21          !     EXCLAMATION MARK (Other_Punctuation)
…

Blocks:

$ uni p arrows 'box drawing'
Showing block Arrows
Showing block Box Drawing
     CPoint  Dec    UTF8        HTML       Name (Cat)
'←'  U+2190  8592   e2 86 90    ←     LEFTWARDS ARROW (Math_Symbol)
'↑'  U+2191  8593   e2 86 91    ↑     UPWARDS ARROW (Math_Symbol)
…

Print as table, and with a shorter name:

$ uni p -as table box
Showing block Box Drawing
         0   1   2   3   4   5   6   7   8   9   A   B   C   D   E   F
       ┌────────────────────────────────────────────────────────────────
U+250x │ ─   ━   │   ┃   ┄   ┅   ┆   ┇   ┈   ┉   ┊   ┋   ┌   ┍   ┎   ┏  
       │
U+251x │ ┐   ┑   ┒   ┓   └   ┕   ┖   ┗   ┘   ┙   ┚   ┛   ├   ┝   ┞   ┟  
       │
U+252x │ ┠   ┡   ┢   ┣   ┤   ┥   ┦   ┧   ┨   ┩   ┪   ┫   ┬   ┭   ┮   ┯  
       │
U+253x │ ┰   ┱   ┲   ┳   ┴   ┵   ┶   ┷   ┸   ┹   ┺   ┻   ┼   ┽   ┾   ┿  
       │
U+254x │ ╀   ╁   ╂   ╃   ╄   ╅   ╆   ╇   ╈   ╉   ╊   ╋   ╌   ╍   ╎   ╏  
       │
U+255x │ ═   ║   ╒   ╓   ╔   ╕   ╖   ╗   ╘   ╙   ╚   ╛   ╜   ╝   ╞   ╟  
       │
U+256x │ ╠   ╡   ╢   ╣   ╤   ╥   ╦   ╧   ╨   ╩   ╪   ╫   ╬   ╭   ╮   ╯  
       │
U+257x │ ╰   ╱   ╲   ╳   ╴   ╵   ╶   ╷   ╸   ╹   ╺   ╻   ╼   ╽   ╾   ╿  
       │

Or more compact table:

$ uni p -as table box -compact
         0   1   2   3   4   5   6   7   8   9   A   B   C   D   E   F
       ┌────────────────────────────────────────────────────────────────
U+250x │ ─   ━   │   ┃   ┄   ┅   ┆   ┇   ┈   ┉   ┊   ┋   ┌   ┍   ┎   ┏  
U+251x │ ┐   ┑   ┒   ┓   └   ┕   ┖   ┗   ┘   ┙   ┚   ┛   ├   ┝   ┞   ┟  
U+252x │ ┠   ┡   ┢   ┣   ┤   ┥   ┦   ┧   ┨   ┩   ┪   ┫   ┬   ┭   ┮   ┯  
U+253x │ ┰   ┱   ┲   ┳   ┴   ┵   ┶   ┷   ┸   ┹   ┺   ┻   ┼   ┽   ┾   ┿  
U+254x │ ╀   ╁   ╂   ╃   ╄   ╅   ╆   ╇   ╈   ╉   ╊   ╋   ╌   ╍   ╎   ╏  
U+255x │ ═   ║   ╒   ╓   ╔   ╕   ╖   ╗   ╘   ╙   ╚   ╛   ╜   ╝   ╞   ╟  
U+256x │ ╠   ╡   ╢   ╣   ╤   ╥   ╦   ╧   ╨   ╩   ╪   ╫   ╬   ╭   ╮   ╯  
U+257x │ ╰   ╱   ╲   ╳   ╴   ╵   ╶   ╷   ╸   ╹   ╺   ╻   ╼   ╽   ╾   ╿  

Emoji

The emoji command (shortcut: e) is is the real reason I wrote this:

$ uni e cry
	Name                     (Cldr)
🥹	face holding back tears  (angry, cry, proud, resist, sad)
😢	crying face              (sad, tear)
😭	loudly crying face       (sad, sob, tear)
😿	crying cat               (face, sad, tear)
🔮	crystal ball             (fairy tale, fantasy, fortune, tool)

By default both the name and CLDR data are searched; the CLDR data is a list of keywords for an emoji; prefix with name: or n: to search on the name only:

$ uni e smile
	Name                             (Cldr)
😃	grinning face with big eyes      (mouth, open, smile)
😄	grinning face with smiling eyes  (mouth, open, smile)
…

$ uni e name:smile
	Name                (Cldr)
😼	cat with wry smile  (face, ironic)

As you can see, the CLDR is pretty useful, as "smile" only gives one result as most emojis use "smiling".

Prefix with group: to search by group:

$ uni e group:hands
	Name               (Cldr)
👏	clapping hands     ()
🙌	raising hands      (celebration, gesture, hooray, raised)
🫶	heart hands        (love)
👐	open hands         ()
🤲	palms up together  (prayer)
🤝	handshake          (agreement, meeting)
🙏	folded hands       (ask, high 5, high five, please, pray, thanks)

Group and search can be combined, and group: can be abbreviated to g::

$ uni e g:cat-face grin
	Name                            (Cldr)
😺	grinning cat                    (face, mouth, open, smile)
😸	grinning cat with smiling eyes  (face, smile)

Like with search, use -or to OR the parameters together instead of AND:

$ uni e -or g:face-glasses g:face-hat
	Name                          (Cldr)
🤠	cowboy hat face               (cowgirl)
🥳	partying face                 (celebration, hat, horn)
🥸	disguised face                (glasses, incognito, nose)
😎	smiling face with sunglasses  (bright, cool)
🤓	nerd face                     (geek)
🧐	face with monocle             (stuffy)

Apply skin tone modifiers with -tone:

$ uni e -tone dark g:hands
	Name                               (Cldr)
👏🏿	clapping hands: dark skin tone     ()
🙌🏿	raising hands: dark skin tone      (celebration, gesture, hooray, raised)
🫶🏿	heart hands: dark skin tone        (love)
👐🏿	open hands: dark skin tone         ()
🤲🏿	palms up together: dark skin tone  (prayer)
🤝	handshake                          (agreement, meeting)
🙏🏿	folded hands: dark skin tone       (ask, high 5, high five, please, pray, thanks)

The "heart hands" may not show as it's very recent. The handshake emoji supports setting individual skin tones per hand since Unicode 14, but this isn't supported, mostly because I can't really really think a good CLI interface for setting this without breaking compatibility (there are some other emojis too, like "holding hands" and "kissing" where you can set both the gender and skin tone of both sides individually). Maybe for uni v3 someday.

The default is to display only the gender-neutral "person", but this can be changed with the -gender option:

$ uni e -gender man g:person-gesture
	Name              (Cldr)
🙍‍♂️	man frowning      (gesture, person frowning)
🙎‍♂️	man pouting       (gesture, person pouting)
🙅‍♂️	man gesturing NO  (forbidden, gesture, hand, person gesturing NO, prohibited)
🙆‍♂️	man gesturing OK  (gesture, hand, person gesturing OK)
💁‍♂️	man tipping hand  (help, information, person tipping hand, sassy)
🙋‍♂️	man raising hand  (gesture, happy, person raising hand, raised)
🧏‍♂️	deaf man          (accessibility, deaf person, ear, hear)
🙇‍♂️	man bowing        (apology, gesture, person bowing, sorry)
🤦‍♂️	man facepalming   (disbelief, exasperation, person facepalming)
🤷‍♂️	man shrugging     (doubt, ignorance, indifference, person shrugging)

Both -tone and -gender accept multiple values. -gender women,man will display both the female and male variants, and -tone light,dark will display both a light and dark skin tone; use all to display all skin tones or genders:

$ uni e -tone light,dark -gender f,m shrug
	Name                              (Cldr)
🤷🏻‍♂️	man shrugging: light skin tone    (doubt, ignorance, indifference, person shrugging)
🤷🏻‍♀️	woman shrugging: light skin tone  (doubt, ignorance, indifference, person shrugging)
🤷🏿‍♂️	man shrugging: dark skin tone     (doubt, ignorance, indifference, person shrugging)
🤷🏿‍♀️	woman shrugging: dark skin tone   (doubt, ignorance, indifference, person shrugging)

Like print and identify, you can use -format:

$ uni e g:cat-face -c -format '%(name): %(emoji)'
grinning cat: 😺
grinning cat with smiling eyes: 😸
cat with tears of joy: 😹
smiling cat with heart-eyes: 😻
cat with wry smile: 😼
kissing cat: 😽
weary cat: 🙀
crying cat: 😿
pouting cat: 😾

See uni help for more details on the -format flag.

JSON

With -as json or -as j you can output the data as JSON:

$ uni i -as json h€ý
[{
	"cat": "Lowercase_Letter",
	"char": "h",
	"cpoint": "U+0068",
	"dec": "104",
	"html": "h",
	"name": "LATIN SMALL LETTER H",
	"utf8": "68"
}, {
	"cat": "Currency_Symbol",
	"char": "€",
	"cpoint": "U+20AC",
	"dec": "8364",
	"html": "€",
	"name": "EURO SIGN",
	"utf8": "e2 82 ac"
}, {
	"cat": "Lowercase_Letter",
	"char": "ý",
	"cpoint": "U+00FD",
	"dec": "253",
	"html": "ý",
	"name": "LATIN SMALL LETTER Y WITH ACUTE",
	"utf8": "c3 bd"
}]

All the columns listed in -f will be included; you can use -f all to include all columns:

$ uni i -as json -f all h€ý
[{
	"bin": "1101000",
	"block": "Basic Latin",
	"cat": "Lowercase_Letter",
	"char": "h",
	"cpoint": "U+0068",
	"dec": "104",
	"digraph": "h",
	"hex": "68",
	"html": "h",
	"json": "\\u0068",
	"keysym": "h",
	"name": "LATIN SMALL LETTER H",
	"oct": "150",
	"plane": "Basic Multilingual Plane",
	"props": "",
	"script": "Latin",
	"unicode": "1.1",
	"utf16be": "00 68",
	"utf16le": "68 00",
	"utf8": "68",
	"width": "neutral",
	"xml": "h"
}, {
	"bin": "10000010101100",
	"block": "Currency Symbols",
	"cat": "Currency_Symbol",
	"char": "€",
	"cpoint": "U+20AC",
	"dec": "8364",
	"digraph": "=e",
	"hex": "20ac",
	"html": "€",
	"json": "\\u20ac",
	"keysym": "EuroSign",
	"name": "EURO SIGN",
	"oct": "20254",
	"plane": "Basic Multilingual Plane",
	"props": "",
	"script": "Common",
	"unicode": "2.1",
	"utf16be": "20 ac",
	"utf16le": "ac 20",
	"utf8": "e2 82 ac",
	"width": "ambiguous",
	"xml": "€"
}, {
	"bin": "11111101",
	"block": "Latin-1 Supplement",
	"cat": "Lowercase_Letter",
	"char": "ý",
	"cpoint": "U+00FD",
	"dec": "253",
	"digraph": "y'",
	"hex": "fd",
	"html": "ý",
	"json": "\\u00fd",
	"keysym": "yacute",
	"name": "LATIN SMALL LETTER Y WITH ACUTE",
	"oct": "375",
	"plane": "Basic Multilingual Plane",
	"props": "",
	"script": "Latin",
	"unicode": "1.1",
	"utf16be": "00 fd",
	"utf16le": "fd 00",
	"utf8": "c3 bd",
	"width": "narrow",
	"xml": "ý"
}]

This also works for the emoji command:

$ uni e -as json -f all 'kissing cat'
[{
	"cldr": "eye, face",
	"cldr_full": "cat, eye, face, kiss, kissing cat",
	"cpoint": "U+1F63D",
	"emoji": "😽",
	"group": "Smileys & Emotion",
	"name": "kissing cat",
	"subgroup": "cat-face"
}]

All values are always a string, even numerical values. This makes things a bit easier/consistent as JSON doesn't support hex literals and such. Use jq or some other tool if you want to process the data further.

ChangeLog

v2.6.0 (2023-11-24)

  • Update to Unicode 15.1.

  • Add "script" property – also supported in the list and print commands:

    % uni identify -f '%(script l:auto) %(cpoint) %(name)' 'a Ω'
    Script CPoint Name
    Latin  U+0061 LATIN SMALL LETTER A
    Common U+0020 SPACE
    Greek  U+03A9 GREEK CAPITAL LETTER OMEGA
    
    % uni list scripts
    Scripts:
    Name                    Assigned
    Adlam                         83
    Ahom                          54
    Anatolian Hieroglyphs        582
    …
    
    % uni print 'script:linear a'
    Showing script Linear A
         CPoint  Dec    UTF8        HTML       Name (Cat)
    '𐘀'  U+10600 67072  f0 90 98 80 𐘀  LINEAR A SIGN AB001 (Other_Letter)
    '𐘁'  U+10601 67073  f0 90 98 81 𐘁  LINEAR A SIGN AB002 (Other_Letter)
    '𐘂'  U+10602 67074  f0 90 98 82 𐘂  LINEAR A SIGN AB003 (Other_Letter)
    …
    
  • Add "unicode" property, which tells you in which Unicode version a codepoint was introduced:

    % uni identify -f '%(unicode l:auto) %(cpoint l:auto) %(name)' a𐘂🫁
    Unicode CPoint  Name
    1.1     U+0061  LATIN SMALL LETTER A
    7.0     U+10602 LINEAR A SIGN AB003
    13.0    U+1FAC1 LUNGS
    
  • Show unprintable control characters as the open box (␣, U+2423) instead of the replacement character (�, U+FFFD). It already did that for C1 control characters, and U+FFFD looked more like a bug than intentional. The -raw/-r flag still overrides this.

  • Always print Private Use characters as-is for %(char) instead of using U+FFFD replacement character. It's usually safe to print this, and having to use -raw is confusing.

  • ls command is now an alias for `list.

2.5.1 (2022-05-09)

  • Fix build on Go 1.17 and earlier.

2.5.0 (2022-05-03)

  • Add support for properties; they can be displayed with %(props) in -format, and selected in print (e.g. uni print dash).

  • Add uni list command, to list categories, blocks, and properties.

  • Allow explicitly selecting a block, category, or property in print with block:name (b:name), category:name (cat:name, c:name), or property:name (prop:name, p:name).

    Also print an error if a string without prefix matched more than one group (i.e. uni p dash matches both the property Dash and category Dash_Punctuation).

  • Add table layout with -as table. Also change -json/-j to -as json or -as j. The -json flag is still accepted as an alias for compatibility.

  • Change -q/-quiet to -c/-compact; -as json will print as minified if given, and -as table will include less padding. -q is still accepted as an alias for compatibility.

  • Don't use the Go stdlib unicode package; since this is a Unicode 13 database and some operations would fail on codepoints added in Unicode 14 due to the mismatch.

v2.4.0 (2021-12-20)

  • Update import path to zgo.at/uni/v2.

  • Add oct and bin flags for -f to print a codepoint as octal or binary.

  • Add f format flag to change the fill character with alignment; e.g. %(bin r:auto f:0) will print zeros on the left.

  • Allow using just o123 for an octal number (instead of 0o123). We can't do this for binary and decimal numbers (since b and d are valid hexidecimals), but no reason not to do it for o.

v2.3.0 (2021-10-05)

  • Update to Unicode 14.0.

  • UTF-16 and JSON are printed as lower case, just like UTF-8 was. Upper-case is used only for codepoints (i.e. U+00AC).

  • uni print can now print from UTF-8 byte sequence; for example to print the € sign:

    uni p utf8:e282ac
    uni p 'utf8:e2 82 ac'
    uni p 'utf8:0xe2 0x82 0xac'
    

    Bytes can optionally be separated by any combination of 0x, -, _, or spaces.

v2.2.1 (2021-06-15)

  • You can now use uni p 0d40 to get U+28 by decimal.

    uni print 40 interprets the 40 as hex instead of decimal, and there was no way to get a codepoint by decimal number. Since codepoints are much more more common than decimals, leaving off the U+ and U is a useful shortcut I'd like to keep. AFAIK there isn't really a standard(-ish) was to explicitly indicate a number is a decimal, so this is probably the closest.

v2.2.0 (2021-06-05)

  • Make proper use of the /v2 import path so that go get and go install work. (#26)

  • Don't panic if -f doesn't contain any formatting characters.

v2.1.0 (2021-03-30)

  • Can now output as JSON with -j or -json.

  • -format all is a special value to include all columns uni knows about. This is useful especially in combination with -json.

  • Add %(block), %(plane), %(width), %(utf16be), %(utf16le), and %(json) to -f`.

  • Refactor the arp242.net/uni/unidata package to be more useful for other use cases. This isn't really relevant for uni users as such, but if you want to get information about codepoints or emojis then this package is a nice addition to the standard library's unicode package.

v2.0.0 (2021-01-03)

This changes some flags, semantics, and defaults in incompatible ways, hence the bump to 2.0. If you use the dmenu-uni script with dmenu or fzf, then you'll need to update that to.

  • Remove the -group flag in favour of group:name syntax; this is more flexible and will allow adding more query syntax later.

    uni emoji -group groupname,othergroup                  Old syntax
    uni emoji -group groupname,othergroup smile            Old syntax
    
    uni emoji -or group:groupname group:othergroup         New syntax
    uni emoji -or group:groupname group:othergroup smile   New syntax
    
    uni emoji -or g:groupname g:othergroup                 Can use shorter g: instead of group:
    
  • Default for -gender is now person instead of all; including all genders by default isn't all that useful, and the gender-neutral "person" should be a fine default for most, just as the skin colour-neutral "yellow" is probably a fine default for most.

  • Add new -or/-o flag. The default for search and emoji is to show everything where all query parameters match ("AND"); with this flag it shows everything where at least one parameter matches ("OR").

  • Add new -format/-f flag to control which columns to output and column width. You can now also print X11 keysyms and Vim digraphs. See uni help for details.

  • Include CLDR data for emojis, which is searched by default if you use uni e <something>. You can use uni e name:x to search for the name specifically.

  • Show a short terse help when using just uni, and a more detailed help on uni help. I hate it when programs print 5 pages of text to my terminal when I didn't ask for it.

  • Update Unicode data to 13.1.

  • Add option to output to $PAGER with -p or -pager. This isn't done automatically (I don't really like it when programs throw me in a pager), but you can define a shell alias (alias uni='uni -p') if you want it by default since flags can be both before or after the command.

v1.1.1 (2020-05-31)

  • Fix tests of v1.1.0, requested by a packager. No changes other than this.

v1.1.0 (2020-03-17)

  • Update Unicode data from 12.1 to 13.0.

  • print command supports codepoints as hex (0xff), octal (0o42), and binary (0b1001).

  • A few very small bugfixes.

v1.0.0 (2019-12-12)

  • Initial release

Development

Re-generate the Unicode data with go generate unidata. Files are cached in unidata/.cache, so clear that if you want to update the files from remote. This requires zsh and GNU awk (gawk).

Alternatives

CLI/TUI

  • https://github.com/philpennock/character

    More or less similar to uni, but very different CLI, and has some additional features. Seems pretty good.

  • https://github.com/sindresorhus/emoj

    Doesn't support emojis sequences (e.g. MAN SHRUGGING is PERSON SHRUGGING + MAN, FIREFIGHTER is PERSON + FIRE TRUCK, etc); quite slow for a CLI program (emoj smiling takes 1.8s on my system, sometimes a lot longer), search results are pretty bad (shrug returns unamused face, thinking face, eyes, confused face, neutral face, tears of joy, and expressionless face ... but not the shrugging emoji), not a fan of npm (has 1862 dependencies).

  • https://github.com/Fingel/tuimoji

    Grouping could be better, doesn't support emojis sequences, only interactive TUI, feels kinda slow-ish especially when searching.

  • https://github.com/pemistahl/chr

    Only deals with codepoints, not emojis.

GUI

  • gnome-characters

    Uses Gnome interface/window decorations and won't work well with other WMs, doesn't deal with emoji sequences, I don't like the grouping/ordering it uses, requires two clicks to copy a character.

  • gucharmap

    Doesn't display emojis, just unicode blocks.

  • KCharSelect

    Many KDE-specific dependencies (106M). Didn't try it.

  • https://github.com/Mange/rofi-emoji and https://github.com/fdw/rofimoji

    Both are pretty similar to the dmenu/rofi integration of uni with some minor differences, and both seem to work well with no major issues.

  • gtk3 emoji picker (Ctrl+; or Ctrl+. in gtk 3.93 or newer)

    Only works in GTK, doesn't work with GTK_IM_MODULE=xim (needed for compose key), for some reasons the emojis look ugly, doesn't display emojis sequences, doesn't have a tooltip or other text description about what the emoji actually is, the variation selector doesn't seem to work (never displays skin tone?), doesn't work in Firefox.

    This is so broken on my system that it seems that I'm missing something for this to work or something?

  • https://github.com/rugk/awesome-emoji-picker

    Only works in Firefox; takes a tad too long to open; doesn't support skin tones.

Didn't investigate (yet)

Some alternatives people have suggested that I haven't looked at; make an issue or email me if you know of any others.

uni's People

Contributors

arp242 avatar esdnm avatar m-cz avatar muesli avatar priner avatar rolandwalker avatar shuuji3 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

uni's Issues

Give `print` an option to not sort characters (or make it not sort by default?)

$ uni print 43 42 41 43
     CPoint  Dec    UTF8        HTML       Name (Cat)
'A'  U+0041  65     41          &#x41;     LATIN CAPITAL LETT… (Uppercase_Let…)
'B'  U+0042  66     42          &#x42;     LATIN CAPITAL LETT… (Uppercase_Let…)
'C'  U+0043  67     43          &#x43;     LATIN CAPITAL LETT… (Uppercase_Let…)
'C'  U+0043  67     43          &#x43;     LATIN CAPITAL LETT… (Uppercase_Let…)

As you can see, the output from print sorts characters by codepoints rather than outputting them in the order given on the command line. This is a problem for scripted use, and it's inconsistent with how uni identify works. It also seems strange that the codepoints are sorted but duplicates are not removed.

Add a -v (version) flag

Hi Martin,

I stumbled upon Uni again, after using it quite awhile ago. I was thinking, I wonder if that's been updated?

But uni -v just returns standard help, and not a release or version number, as per convention.

So this issue serves to say, -v is useful to users who come back to uni after moderate period of time.

I may submit a PR for this, but just leaving this here in the meantime. Proost!

FR: use CLDR Character Annotation “Keywords” too when searching characters/emoji

Unicode CLDR Character Annotation has provided a list of keywords for some characters (especially emoji) that is to enhance the search experience of them.

The remaining phrases are keywords (labels), separated by “|”. The keywords plus the words in the short name are typically used for search and predictive typing.
— CLDR Character Annotations description

I would like to suggest to include these keywords too when searching for both Unicode characters and emojis.

A List of these annotations can be found here:
https://www.unicode.org/cldr/charts/36/annotations/romance.html

Computer-friendly character annotation data in XML for each language can be found here: https://github.com/unicode-org/cldr/tree/master/common/annotations

Support listing by derived properties

Unicode has a concept of "derived properties" (the canonical list is here) which are defined as a combination of other character classes. For example, the first entry in that list is:

# Derived Property: Math
#  Generated from: Sm + Other_Math

It would be nice to be able to pass these as arguments to print.

allow search to return all

I'd prefer to use search for listing emojis, as it includes, just for example what I was missing now: '☐' U+2610 9744 e2 98 90 &#x2610; BALLOT BOX (Other_Symbol). emoji does not include this, because it is not deemed an emoji, not in certain emoji codepoint blocks, or so?

Test failures

With the new release (or current master), I get the following tests failures:

--- FAIL: TestPrint (0.11s)
    --- FAIL: TestPrint/[-q_p_OtherPunctuation] (0.00s)
        uni_test.go:144: wrong # of lines
            out:  593
            want: 588
    --- FAIL: TestPrint/[-q_p_Po] (0.00s)
        uni_test.go:144: wrong # of lines
            out:  593
            want: 588
    --- FAIL: TestPrint/[-q_p_all] (0.11s)
        uni_test.go:144: wrong # of lines
            out:  33797
            want: 32841
--- FAIL: TestEmoji (0.01s)
    --- FAIL: TestEmoji/[e_-tone_mediumlight_bride] (0.00s)
        uni_test.go:217: wrong output
            out:  []string{""}
            want: []string{"👰🏼"}
--- FAIL: TestAllEmoji (0.01s)
    uni_test.go:248: different length: want 3078, got 3195
    uni_test.go:252: 
FAIL
FAIL	arp242.net/uni	0.160s
ok  	arp242.net/uni/isatty	0.004s
?   	arp242.net/uni/terminal	[no test files]
?   	arp242.net/uni/unidata	[no test files]
FAIL

identify: Uni is discarding Space, Tab and Newline characters

$ printf "a b\tc\n" | hexyl 
┌────────┬─────────────────────────┬─────────────────────────┬────────┬────────┐
│00000000│ 61 20 62 09 63 0a       ┊                         │a b_c_  ┊        │
└────────┴─────────────────────────┴─────────────────────────┴────────┴────────┘

$ printf "a b\tc\n" | uni i
     CPoint  Dec    UTF8        HTML       Name (Cat)
'a'  U+0061  97     61          &#x61;     LATIN SMALL LETTER A (Lowercase_Letter)
'b'  U+0062  98     62          &#x62;     LATIN SMALL LETTER B (Lowercase_Letter)
'c'  U+0063  99     63          &#x63;     LATIN SMALL LETTER C (Lowercase_Letter)


Custom columns

Thanks for writing uni, I find it very useful!

I've found myself trying to make an easy reference to Unicode-related data over time, such as

  • Vim digraphs
  • X11 keysyms

and more will probably be added in the future. I could put each one in Markdown format but it would be nice if they could integrate with uni.

Hardcoding these into the program itself would probably be feature creep, but a way to integrate different data would be really useful.

Private use characters are output as replacement characters

When I use uni to query information on a private use character — such as the Apple logo, defined as U+F8FF on Apple computers — uni actually outputs U+FFFD, the replacement character. For example:

$ uni identify 
     CPoint  Dec    UTF8        HTML       Name (Cat)
'�'  U+F8FF  63743  ef a3 bf    &#xf8ff;   <Private Use, Last> (Private_Use)

The command above ends with U+F8FF, but the character output by uni is . Evidence:

$ uni identify  -c -f '%(char)' | hexdump -C
00000000  ef bf bd 0a                                       |....|
00000004

ef bf bd is the UTF-8 encoding for U+FFFD, not for U+F8FF.

`uni list` - `from` and `to` are identical

I have a strong suspicion the error happens here -

uni/uni.go

Lines 531 to 532 in 7084420

"from": fmt.Sprintf(fmtCp, b.Range[0]),
"to": fmt.Sprintf(fmtCp, b.Range[0]),
and should be b.range[1] - but I don't know enough about go to be sure and PR it.

Thank you

Show character codes in emoji mode

Thanks for an excellent tool!

I wish there were a direct way to show character codes in emoji mode. Right now we have:

% uni e switzerland
🇨🇭 flag: Switzerland  Flags  country-flag

and I have to copy & paste that character to uni i. Maybe add a verbose flag to get:

% uni e --verbose switzerland
🇨🇭 flag: Switzerland  Flags  country-flag
    '🇨'  U+1F1E8 127464 f0 9f 87 a8 &#x1f1e8;  REGIONAL INDICATOR SYMBOL LETTER C (Other_Symbol)
    '🇭'  U+1F1ED 127469 f0 9f 87 ad &#x1f1ed;  REGIONAL INDICATOR SYMBOL LETTER H (Other_Symbol)

FZF integration

Adding an integration to FZF could make this tool replace emoji-fzf. I'm up for implementing this and opening a PR after exams finish.

Make v2 installable via "go install"

Currently, go install arp242.net/uni@latest installs the latest 1.x version. Also, https://pkg.go.dev/arp242.net/uni claims version 1.1.1 is the latest.

With the way Go module versioning works, I believe the module name should be updated to arp242.net/uni/v2 now, and this should be reflected in the go.mod. Possibly https://arp242.net/uni/v2 also needs to be enabled, not sure.

See https://golang.org/doc/modules/major-version, https://golang.org/doc/modules/release-workflow#breaking.

Search by utf8?

Recently had a problem with some code I copied from a coworker from Slack. For some reason, lines showed up as having been changed in git even though I couldn't see what was different. Put it through a hex editor and saw e2 80 8b. Went to my usual tool for this type of thing, FileFormat.info, typed that in, and it came up with the right answer, that there were zero-width spaces inserted.

I'd like to be able to use uni to search by utf8 text like that.

Possible to add support for detecting fonts containing the specified character?

This might be outside the scope of what you have in mind, and there may not be sufficient libraries in Go to make this something that is possible to implement in a reasonable amount of time (and it would probably only work on linux / require different solutions for different platforms), but... It would be really nice if it were possible to also detect fonts that contain the queried character, e.g. something like the perl example from this Polybar wiki:

use strict;
use warnings;
use Font::FreeType;
my ($char) = @ARGV;
foreach my $font_def (`fc-list`) {
    my ($file, $name) = split(/: /, $font_def);
    my $face = Font::FreeType->new->face($file);
    my $glyph = $face->glyph_from_char($char);
    if ($glyph) {
        print $font_def;
    }
}

Just a thought..

Allow copying to clipboard

From HN comment https://news.ycombinator.com/item?id=21780608

echo -n "$selected_symbol" | xclip -i

And then fake a middle click to insert in the current application:

xdotool click 2o

Also maybe send as direct input (xdotool can do this, but is apparently not reliable across all apps/characters).

Can also link to libX11, but this requires cgo and makes builds harder.

Give control over which variation selectors to print

The U+FE0F and U+FE0E can control if a character is displayed in "character style" or "emoji style"; this matters for some characters that have representations in both:

↔︎ (fe0e, text)
↔️ (fe0f, emoji)

These things are kind "hidden" and tricky, because many editors don't display them at all, including Vim, and different environments have different defaults. I lost quite a bit of time tracking down a problem with this a few weeks ago :-/

Either way, there should be a flag for the emoji command to say which variation selector to use. This also applies to the other commands, but those all assume there is just one codepoint now in various columns, so meh.

Zip with Windows release

I see here

https://github.com/arp242/uni/releases

that all releases are GZ, even Windows. This isnt a huge deal, as you can get
tools to extract GZ on Windows. However to my knowledge they arent built in,
even with Windows 10.

So it might make sense to offer ZIP builds instead to Windows users.

Use embeded SQLite?

Just a random idea I'll jot down: what if instead of using Go structs in unidata for all the codepoints etc. we would use a SQLite database compiled in the binary?

This would be more structured, easier to reuse, and most of all: easy to query. While the current CLI is convenient, sometimes you want to know things like "which characters are full-width"?

Need to look at the feasibility of this, and how it will affect performance, binary size, etc. Also SQLite requires cgo which makes the builds a bit harder, but this may be a nice use case for e.g. https://modernc.org/sqlite

Add aarch64 and arm binaries for Termux

Package: golang
Version: 2:1.13.5
Maintainer: Fredrik Fornwall @fornwall
Installed-Size: 325 MB
Depends: clang
Homepage: https://golang.org/
Download-Size: 53.3 MB
APT-Sources: https://termux.org/packages stable/main aarch64 Packages
Description: Go programming language compiler

Go lang is 325 MB and I don't have that much free space(My Termux has already occupied 3GB 😢 ), it would be great if you add a aarch64 & arm binaries. :)

clear whitespace at start of args

" search euro" verus "search euro" gives differernt results.

Basically just need to remove white space from start of args. Simple one, but makes things a little less error prone.
Wonder if fuzzing would have picked this up ? anyways


$ uni  search euro
Usage: uni [command] [flags]

uni queries the unicode database. https://github.com/arp242/uni

Flags:
    -f, -format    Output format.
    -a, -as        How to print the results: list (default), json, or table.
    -c, -compact   More compact output.
    -r, -raw       Don't use graphical variants or add combining characters.
    -p, -pager     Output to $PAGER.
    -o, -or        Use "or" when searching instead of "and".

Commands:
    list           List blocks, categories, or properties.
    identify       Identify all the characters in the given strings.
    search         Search description for any of the words.
    print          Print characters by codepoint, category, or block.
    emoji          Search emojis.

Use "uni help" or "uni -h" for a more detailed help.
$ uni search euro
     CPoint  Dec    UTF8        HTML       Name (Cat)
'₠'  U+20A0  8352   e2 82 a0    &#x20a0;   EURO-CURRENCY SIGN (Currency_Symbol)
'€'  U+20AC  8364   e2 82 ac    &euro;     EURO SIGN (Currency_Symbol)
'𐡷'  U+10877 67703  f0 90 a1 b7 &#x10877;  PALMYRENE LEFT-POINTING FLEURON (Other_Symbol)
'𐡸'  U+10878 67704  f0 90 a1 b8 &#x10878;  PALMYRENE RIGHT-POINTING FLEURON (Other_Symbol)
'𐫱'  U+10AF1 68337  f0 90 ab b1 &#x10af1;  MANICHAEAN PUNCTUATION FLEURON (Other_Punctuation)
'🌍' U+1F30D 127757 f0 9f 8c 8d &#x1f30d;  EARTH GLOBE EUROPE-AFRICA (Other_Symbol)
'🏤' U+1F3E4 127972 f0 9f 8f a4 &#x1f3e4;  EUROPEAN POST OFFICE (Other_Symbol)
'🏰' U+1F3F0 127984 f0 9f 8f b0 &#x1f3f0;  EUROPEAN CASTLE (Other_Symbol)
'💶' U+1F4B6 128182 f0 9f 92 b6 &#x1f4b6;  BANKNOTE WITH EURO SIGN (Other_Symbol)

installation fails with " t.Cleanup undefined (type *testing.T has no field or method Cleanup)"

hi :)

Env

go version go1.13.8 linux/amd64

Bug

running go build produces the following output and exits with a non-zero code

go: downloading zgo.at/zstd v0.0.0-20210322015326-ca7824321150
go: downloading zgo.at/zli v0.0.0-20210330134141-b5f2a73532d6
go: extracting zgo.at/zstd v0.0.0-20210322015326-ca7824321150
go: extracting zgo.at/zli v0.0.0-20210330134141-b5f2a73532d6
go: downloading golang.org/x/term v0.0.0-20210317153231-de623e64d2a6
go: extracting golang.org/x/term v0.0.0-20210317153231-de623e64d2a6
go: downloading golang.org/x/sys v0.0.0-20210315160823-c6e025ad8005
go: extracting golang.org/x/sys v0.0.0-20210315160823-c6e025ad8005
go: finding zgo.at/zstd v0.0.0-20210322015326-ca7824321150
go: finding zgo.at/zli v0.0.0-20210330134141-b5f2a73532d6
go: finding golang.org/x/term v0.0.0-20210317153231-de623e64d2a6
go: finding golang.org/x/sys v0.0.0-20210315160823-c6e025ad8005
# zgo.at/zli
../../.go/pkg/mod/zgo.at/[email protected]/test.go:84:3: t.Cleanup undefined (type *testing.T has no field or method Cleanup)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.