fumiyas / libiconv-utf8mac Goto Github PK

View Code? Open in Web Editor NEW

16.0 7.0 1.0 9.23 MB

GNU libiconv with UTF-8-MAC support (Port from Apple's GNU libiconv)

License: GNU General Public License v3.0

Makefile 1.53% Shell 11.10% C 83.96% M4 0.28% Batchfile 0.02% sed 0.01% C++ 0.08% Roff 3.01%

unicode iconv macos encoding utf-8-mac nfd

libiconv-utf8mac's Introduction

GNU libiconv with UTF-8-MAC support (Port from Apple's GNU libiconv)

Copyright (c) 2018-2022 SATOH Fumiyasu @ OSS Technology Corp., Japan
License: GNU General Public License Version 3 (iconv program and docs), GNU Library General Public License Version 2 (libraries)
Development home: https://github.com/fumiyas/libiconv-utf8mac
Author's home: https://fumiyas.github.io/

Works

Base source: GNU libiconv 1.17
UTF-8-MAC support: Apple libiconv-51.200.6 (utf8mac.h)
My UTF-8-MAC works:
- Port to Linux, Solaris, AIX
- Support surrogate pairs
- Fix bugs

Building libiconv program and libraries from the Git repository

On Debian or Ubuntu system:

$ sudo apt install gcc make autoconf automake gperf gettext groff gnulib git
...
$ git clone https://github.com/fumiyas/libiconv-utf8mac.git
...
$ cd libiconv-utf8mac
$ git rev-parse --abbrev-ref @ # Note: Check if the current tree is `utf-8-mac-*`
utf-8-mac-51.200.6.libiconv-1.17
$ make -f Makefile.utf8mac autogen
...
$ ./configure
...
$ make
...
$ sudo make install
...
$ /usr/local/bin/iconv -l |grep UTF-8-MAC
UTF-8-MAC UTF8-MAC

Make a libiconv source tar ball

$ make -f Makefile.utf8mac dist
...

Note

UTF-8-MAC is a NFD-like normalization form, not the NFD!!!!

Original	UTF-8-MAC	NFD
福 (U+FA1B)	←	福 (U+798F)
神 (U+FA19)	←	神 (U+795E)
づ (U+3065)	づ (U+3064 U+3099)	←
け (U+3051)	←	←

TODO

Add UTF-8-NOMAC encoding (NFC-like normalization form)

libiconv-utf8mac's People

Contributors

Stargazers

Watchers

Forkers

noloader

libiconv-utf8mac's Issues

fatal error: byteswap.h: No such file or directory

Hi Everyone,

I'm testing on FreeBSD 12.0, x86_64. One compile error:

libtool: compile:  gcc -I. -I. -I../include -I./../include -I.. -I./.. -I/home/jwalton/tmp/ok2delete/include -DNDEBUG -g2 -O2 -march=native -fPIC -pthread -fvisibility=hidden -DLIBDIR=\"/home/jwalton/tmp/ok2delete/lib\" -DBUILDING_LIBICONV -DBUILDING_DLL -DENABLE_RELOCATABLE=1 -DIN_LIBRARY -DINSTALLDIR=\"/home/jwalton/tmp/ok2delete/lib\" -DNO_XMALLOC -Dset_relocation_prefix=libiconv_set_relocation_prefix -Drelocate=libiconv_relocate -Drelocate2=libiconv_relocate2 -DHAVE_CONFIG_H -c ./../libcharset/lib/localcharset.c  -fPIC -DPIC -o .libs/localcharset.o
In file included from ./converters.h:125,
                 from ./iconv.c:71:
./utf8mac.h:48:11: fatal error: byteswap.h: No such file or directory
   48 | # include <byteswap.h>
      |           ^~~~~~~~~~~~
compilation terminated.
gmake[1]: *** [Makefile:82: iconv.lo] Error 1

No rule to make target 'build-aux/compile', needed by 'distdir'

Hi Everyone,

I'm trying to build a new tarball according to the README. Most goes well, but the end of make -f Makefile.utf8mac dist fails with:

rm -rf $tmpdistdir $tmpdistdir.tar $tmpdistdir.tar.gz \
  && mkdir $tmpdistdir \
  && make distdir distdir="$abstmpdistdir" \
  && tar chof $tmpdistdir.tar --owner=root --group=root $tmpdistdir \
  && gzip -9 $tmpdistdir.tar \
  && rm -rf $tmpdistdir
make[2]: Entering directory '/home/jwalton/libiconv-utf8mac/libiconv'
make[2]: *** No rule to make target 'build-aux/compile', needed by 'distdir'.  Stop.
make[2]: Leaving directory '/home/jwalton/libiconv-utf8mac/libiconv'
Makefile:241: recipe for target 'dist' failed
make[1]: *** [dist] Error 2
make[1]: Leaving directory '/home/jwalton/libiconv-utf8mac/libiconv'
Makefile.utf8mac:24: recipe for target 'dist' failed

I'm experiencing the problem on both Fedora and Ubuntu. Looking at build-aux, there is ltmain.sh and mkinstalldirs, but there is no compile.

The best I can tell the only difference between our repos are the fixes for the memory leaks.

Resource leaks in libiconv 1.16

Hi Everyone,

Thanks for the port with UTF-8-Mac. I think I may need to switch to it (and move away from the crippled GNU version).

You might consider adding this patch to your version of libiconv: Noloader | libiconv.patch. It cleans up resource leaks in libiconv so the library can get through acceptance testing and security testing and evaluation (ST&E).

Without the patch, both libiconv and programs that depend upon iconv fail to get through testing.

Applying the patch should be as simple as the following:

$ git clone ...
$ cd libiconv-utf8mac/
$ git checkout utf-8-mac-51.200.6.libiconv-1.16
$ wget https://raw.githubusercontent.com/noloader/Build-Scripts/master/patch/iconv.patch
$ patch -p0 < iconv.patch

utf8mac.h:63:27: error: unknown type name 'u_int16_t'

Hi Everyone,

I'm testing on Apline Linux. Alpine's claim to fame is it uses Musl rather then glibc.

The build is failing with:

libtool: compile:  gcc -I. -I. -I../include -I./../include -I.. -I./.. -I/home/jwalton/tmp/ok2delete/include -DNDEBUG -g2 -O2 -march=native -fPIC -pthread -fvisibility=hidden "-DLIBDIR=\"/home/jwalton/tmp/ok2delete/lib\"" -DBUILDING_LIBICONV -DBUILDING_DLL -DENABLE_RELOCATABLE=1 -DIN_LIBRARY "-DINSTALLDIR=\"/home/jwalton/tmp/ok2delete/lib\"" -DNO_XMALLOC -Dset_relocation_prefix=libiconv_set_relocation_prefix -Drelocate=libiconv_relocate -Drelocate2=libiconv_relocate2 -DHAVE_CONFIG_H -c ./../libcharset/lib/localcharset.c  -fPIC -DPIC -o .libs/localcharset.o
In file included from ./converters.h:125,
                 from ./iconv.c:71:
./utf8mac.h:63:27: error: unknown type name 'u_int16_t'
 int utf8_encodestr (const u_int16_t *, size_t, u_int8_t *, size_t *,
                           ^~~~~~~~~
./utf8mac.h:63:48: error: unknown type name 'u_int8_t'; did you mean 'uint8_t'?
 int utf8_encodestr (const u_int16_t *, size_t, u_int8_t *, size_t *,
                                                ^~~~~~~~
                                                uint8_t
./utf8mac.h:64:11: error: unknown type name 'u_int16_t'; did you mean 'uint16_t'?
   size_t, u_int16_t, int);
           ^~~~~~~~~
           uint16_t
./utf8mac.h:66:27: error: unknown type name 'u_int8_t'
 int utf8_decodestr (const u_int8_t *, size_t, u_int16_t *,size_t *,
                           ^~~~~~~~
./utf8mac.h:66:47: error: unknown type name 'u_int16_t'; did you mean 'uint16_t'?
 int utf8_decodestr (const u_int8_t *, size_t, u_int16_t *,size_t *,
                                               ^~~~~~~~~
                                               uint16_t
./utf8mac.h:67:11: error: unknown type name 'u_int16_t'; did you mean 'uint16_t'?
   size_t, u_int16_t, int, size_t *);
           ^~~~~~~~~
           uint16_t
./utf8mac.h:134:14: error: unknown type name 'u_int16_t'
 static const u_int16_t
              ^~~~~~~~~
./utf8mac.h:381:14: error: unknown type name 'u_int32_t'
 static const u_int32_t __UniCharDecompositionTableLength =
              ^~~~~~~~~
./utf8mac.h:382:51: error: 'u_int16_t' undeclared here (not in a function); did you mean 'uint16_t'?
  (sizeof(__CFUniCharDecompositionTable) / (sizeof(u_int16_t) * 2));
                                                   ^~~~~~~~~
                                                   uint16_t
./utf8mac.h:386:1: error: expected '=', ',', ';', 'asm' or '__attribute__' before '__CFUniCharMultipleDecompositionTable'
 __CFUniCharMultipleDecompositionTable[] = {
 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./utf8mac.h:625:14: error: unknown type name 'u_int8_t'
 static const u_int8_t
              ^~~~~~~~
./utf8mac.h:734:14: error: unknown type name 'u_int32_t'
 static const u_int32_t
              ^~~~~~~~~
./utf8mac.h:765:14: error: unknown type name 'u_int32_t'
 static const u_int32_t __CFUniCharPrecompositionTableLength =
              ^~~~~~~~~
./utf8mac.h:766:52: error: 'u_int32_t' undeclared here (not in a function); did you mean 'uint32_t'?
   (sizeof(__CFUniCharPrecompSourceTable) / (sizeof(u_int32_t) * 2));
                                                    ^~~~~~~~~
                                                    uint32_t
./utf8mac.h:770:1: error: expected '=', ',', ';', 'asm' or '__attribute__' before '__CFUniCharBMPPrecompDestinationTable'
 __CFUniCharBMPPrecompDestinationTable[] = {
 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./utf8mac.h:1003:14: error: unknown type name 'u_int8_t'
 static const u_int8_t
              ^~~~~~~~
./utf8mac.h:1137:20: error: expected declaration specifiers or '...' before 'u_int16_t'
 unicode_combinable(u_int16_t character)
                    ^~~~~~~~~
./utf8mac.h:1162:23: error: expected declaration specifiers or '...' before 'u_int16_t'
 unicode_decomposeable(u_int16_t character) {
                       ^~~~~~~~~
./utf8mac.h:1180:30: error: expected declaration specifiers or '...' before 'u_int16_t'
 static int unicode_decompose(u_int16_t character, u_int16_t *convertedChars);
                              ^~~~~~~~~
./utf8mac.h:1180:51: error: expected declaration specifiers or '...' before 'u_int16_t'
 static int unicode_decompose(u_int16_t character, u_int16_t *convertedChars);
                                                   ^~~~~~~~~
./utf8mac.h:1181:18: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'unicode_combine'
 static u_int16_t unicode_combine(u_int16_t base, u_int16_t combining);
                  ^~~~~~~~~~~~~~~
./utf8mac.h:1206:22: warning: type defaults to 'int' in declaration of 'u_int16_t' [-Wimplicit-int]
 utf8_encodestr(const u_int16_t * ucsp, size_t ucslen, u_int8_t * utf8p,
                      ^~~~~~~~~
./utf8mac.h:1206:32: error: expected ';', ',' or ')' before '*' token
 utf8_encodestr(const u_int16_t * ucsp, size_t ucslen, u_int8_t * utf8p,
                                ^
./utf8mac.h:1318:22: error: unknown type name 'u_int8_t'
 utf8_decodestr(const u_int8_t* utf8p, size_t utf8len, u_int16_t* ucsp,
                      ^~~~~~~~
./utf8mac.h:1318:55: error: expected declaration specifiers or '...' before 'u_int16_t'
 utf8_decodestr(const u_int8_t* utf8p, size_t utf8len, u_int16_t* ucsp,
                                                       ^~~~~~~~~
./utf8mac.h:1319:47: error: expected declaration specifiers or '...' before 'u_int16_t'
                size_t *ucslen, size_t buflen, u_int16_t altslash, int flags, size_t *consumed)
                                               ^~~~~~~~~
./utf8mac.h:1460:2: error: expected specifier-qualifier-list before 'u_int32_t'
  u_int32_t _key;
  ^~~~~~~~~
./utf8mac.h:1465:1: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'getmappedvalue32'
 getmappedvalue32(const unicode_mappings32 *theTable, u_int32_t numElem,
 ^~~~~~~~~~~~~~~~
./utf8mac.h:1488:2: error: expected specifier-qualifier-list before 'u_int16_t'
  u_int16_t _key;
  ^~~~~~~~~
./utf8mac.h:1493:1: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'getmappedvalue16'
 getmappedvalue16(const unicode_mappings16 *theTable, u_int32_t numElem,
 ^~~~~~~~~~~~~~~~
/bin/sh ../libtool --mode=compile gcc -I. -I. -I../include -I./../include -I.. -I./.. -I/home/jwalton/tmp/ok2delete/include -DNDEBUG -g2 -O2 -march=native -fPIC -pthread -fvisibility=hidden -DLIBDIR=\"/home/jwalton/tmp/ok2delete/lib\" -DBUILDING_LIBICONV -DBUILDING_DLL -DENABLE_RELOCATABLE=1 -DIN_LIBRARY -DINSTALLDIR=\"/home/jwalton/tmp/ok2delete/lib\" -DNO_XMALLOC -Dset_relocation_prefix=libiconv_set_relocation_prefix -Drelocate=libiconv_relocate -Drelocate2=libiconv_relocate2 -DHAVE_CONFIG_H -c ./relocatable.c
./utf8mac.h:1517:1: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'unicode_recursive_decompose'
 unicode_recursive_decompose(u_int16_t character, u_int16_t *convertedChars)
 ^~~~~~~~~~~~~~~~~~~~~~~~~~~
./utf8mac.h:1575:19: error: expected declaration specifiers or '...' before 'u_int16_t'
 unicode_decompose(u_int16_t character, u_int16_t *convertedChars)
                   ^~~~~~~~~
./utf8mac.h:1575:40: error: expected declaration specifiers or '...' before 'u_int16_t'
 unicode_decompose(u_int16_t character, u_int16_t *convertedChars)
                                        ^~~~~~~~~
./utf8mac.h:1608:1: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'unicode_combine'
 unicode_combine(u_int16_t base, u_int16_t combining)
 ^~~~~~~~~~~~~~~
./utf8mac.h: In function 'utf8mac_mbtowc':
./utf8mac.h:1648:14: error: expected ';' before 'ucsp'
     u_int16_t ucsp[13];
              ^~~~~
              ;
./utf8mac.h:1653:12: error: 'ucsp' undeclared (first use in this function); did you mean 'bcmp'?
     bzero(&ucsp, sizeof(ucsp));
            ^~~~
            bcmp
./utf8mac.h:1653:12: note: each undeclared identifier is reported only once for each function it appears in
./utf8mac.h:1661:11: warning: implicit declaration of function 'utf8_decodestr' [-Wimplicit-function-declaration]
     ret = utf8_decodestr(s, n, ucsp, &ucslen, sizeof(ucsp), 0, flags, &consumed);
           ^~~~~~~~~~~~~~
./utf8mac.h: In function 'utf8mac_wctomb':
./utf8mac.h:1680:14: error: expected ';' before 'ucs_string'
     u_int16_t ucs_string[13];
              ^~~~~~~~~~~
              ;
./utf8mac.h:1683:49: error: 'ucs_string' undeclared (first use in this function); did you mean 'wcsstr'?
     if((ret = _utf16be_wctomb((unsigned char *) ucs_string, wc, sizeof(ucs_string))) < 0)
                                                 ^~~~~~~~~~
                                                 wcsstr
./utf8mac.h:1690:11: warning: implicit declaration of function 'utf8_encodestr' [-Wimplicit-function-declaration]
     ret = utf8_encodestr(ucs_string, ret, r, &len, n, 0, flags);
           ^~~~~~~~~~~~~~
libtool: compile:  gcc -I. -I. -I../include -I./../include -I.. -I./.. -I/home/jwalton/tmp/ok2delete/include -DNDEBUG -g2 -O2 -march=native -fPIC -pthread -fvisibility=hidden "-DLIBDIR=\"/home/jwalton/tmp/ok2delete/lib\"" -DBUILDING_LIBICONV -DBUILDING_DLL -DENABLE_RELOCATABLE=1 -DIN_LIBRARY "-DINSTALLDIR=\"/home/jwalton/tmp/ok2delete/lib\"" -DNO_XMALLOC -Dset_relocation_prefix=libiconv_set_relocation_prefix -Drelocate=libiconv_relocate -Drelocate2=libiconv_relocate2 -DHAVE_CONFIG_H -c ./relocatable.c  -fPIC -DPIC -o .libs/relocatable.o
make[1]: *** [Makefile:82: iconv.lo] Error 1

Thanks a lot for this

Using your enhancement of utfmac.h, i could finally compile an rsync for mac that does not fail when Umlauts and Emojis are mixed in a filename.
You're my personal hero now 👍

Build fails on Ubuntu 20.04.1

/home/foobar/libiconv-utf8mac-fumiyas/gnulib/gnulib-tool: *** minimum supported autoconf version is 2.64. Try adding AC_PREREQ([2.64]) to your configure.ac.
/home/fooar/libiconv-utf8mac-fumiyas/gnulib/gnulib-tool: *** Stop.
make: *** [Makefile.utf8mac:31: configure] Error 1

make -f Makefile.utf8mac dist fails on Ubuntu 18.04

I'm trying to build a release tarball (hint, hint), and it is failing on Ubuntu 18.04:

$ git branch
* utf-8-mac-51.200.6.libiconv-1.16
$ make -f Makefile.utf8mac dist
...
test -d '/home/jwalton/libiconv-utf8mac/libiconv-1.16.utf8mac51.0.20191219134647.gitcf109ff'/libcharset || mkdir '/home/jwalton/libiconv-utf8mac/libiconv-1.16.utf8mac51.0.20191219134647.gitcf109ff'/libcharset; cd libcharset && make distdir distdir='/home/jwalton/libiconv-utf8mac/libiconv-1.16.utf8mac51.0.20191219134647.gitcf109ff'/libcharset
make[3]: Entering directory '/home/jwalton/libiconv-utf8mac/libcharset'
make[3]: *** No rule to make target 'distdir'.  Stop.
make[3]: Leaving directory '/home/jwalton/libiconv-utf8mac/libcharset'
Makefile:213: recipe for target 'distdir' failed
make[2]: *** [distdir] Error 2
make[2]: Leaving directory '/home/jwalton/libiconv-utf8mac'
Makefile:241: recipe for target 'dist' failed
make[1]: *** [dist] Error 2
make[1]: Leaving directory '/home/jwalton/libiconv-utf8mac'
Makefile.utf8mac:24: recipe for target 'dist' failed
make: *** [dist] Error 2

Also, there is no make -f Makefile.utf8mac clean or make -f Makefile.utf8mac distclean recipes, so I don't know how to clean and start over.

I think it would be really helpful if you released a tarball. Preferably with the patch applied from Issue #1.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.