voku / portable-ascii Goto Github PK

Example: Is this correct?

"besøgende" -> "besoegende"

Can a native speaker, please check the character-replacement, thanks.

https://github.com/voku/portable-ascii/blob/master/src/voku/helper/data/ascii_by_languages.php#L589

Example: Is this correct?

"उदाहरण" -> "uThaHaRaNae"

Can a native speaker, please check the character-replacement, thanks.

https://github.com/voku/portable-ascii/blob/master/src/voku/helper/data/ascii_by_languages.php#L118

Example: Is this correct?

"Odwiedzający" -> "Odwiedzajacy"

Can a native speaker, please check the character-replacement, thanks.

https://github.com/voku/portable-ascii/blob/master/src/voku/helper/data/ascii_by_languages.php#L600

Example: Is this correct?

"ბავშვები" -> "bavshvebi"

Can a native speaker, please check the character-replacement, thanks.

https://github.com/voku/portable-ascii/blob/master/src/voku/helper/data/ascii_by_languages.php#L431

Example: Is this correct?

"müştəri" -> "musteri"

Can a native speaker, please check the character-replacement, thanks.

https://github.com/voku/portable-ascii/blob/master/src/voku/helper/data/ascii_by_languages.php#L1001

Example: Is this correct?

"univerzitní" -> "univerzitni"

Can a native speaker, please check the character-replacement, thanks.

https://github.com/voku/portable-ascii/blob/master/src/voku/helper/data/ascii_by_languages.php#L567

What is this feature about (expected vs actual behaviour)?

Seems like release v1.6.0 broke Laravel's build and subsequently all PR's that are sent in since are failing as well: https://github.com/laravel/framework/runs/4920401730?check_suite_focus=true#step:8:138

It seems most likely that this commit is the culprit: acaf868

Changing the behaviour here seems like a breaking change to me and should probably be done in a major release?

How can I reproduce it?

voku\helper\ASCII::to_ascii('пиздюк.txt');

Expected output: pizdyuk.txt (pre 1.6.0)
Given output: pizdiuk.txt (as of 1.6.0)

Does it take minutes, hours or days to fix?

Think this is an easy fix by just rolling back the commit and tagging a new patch release.

/build is missing :)

Thank you for maintaining this file

Example: Is this correct?

"päiväkoti" -> "paivakoti"

Can a native speaker, please check the character-replacement, thanks.

https://github.com/voku/portable-ascii/blob/master/src/voku/helper/data/ascii_by_languages.php#L424

What is this feature about (expected vs actual behaviour)?

MacOS filesystem replaces German (single character) umlaute with a two characters since Big Sur or only on M1 machines.
These are replaced with a simple a instead of ae. Expected behavior would be to replace these combined characters the same way as the single character umlaute are treated.

How can I reproduce it?

• create a new file or folder with ä, ö or ü - persist the filename (ENTER)
• open the filename change dialog and copy the shown ä/ö/ü character
• run it through the ASCII class

The real-world scenario would be to sanitize an uploaded filename.

Does it take minutes, hours or days to fix?

The local fix took minutes (added a str_replace() before running the ASCII conversion) but no idea how much it will take to properly add it to the package.

Any additional information?

In hope that GitHub doesn't replace the characters - here it is ä - the codepoints are U+61 U+308.
https://www.fileformat.info/info/unicode/char/61/index.htm
https://www.fileformat.info/info/unicode/char/308/index.htm

I'm not sure if this would be the expected behavior of the package or I want it to do something that's not in the scope of that package. We have a working solution right now - so it's in no way urgent.

Example: Is this correct?

"Ülikool" -> "Ulikool"

Can a native speaker, please check the character-replacement, thanks.

https://github.com/voku/portable-ascii/blob/master/src/voku/helper/data/ascii_by_languages.php#L653

Example: Is this correct?

"infanĝardeno" -> "infangxardeno"

Can a native speaker, please check the character-replacement, thanks.

https://github.com/voku/portable-ascii/blob/master/src/voku/helper/data/ascii_by_languages.php#L637

The recent release growed in size by ~45000 lines because of 5417140

We bundle the lib with our cms and it makes a big portion of our overall application size, see redaxo/redaxo#3662

Do you have any recommendations/suggestions on how we could save in filesize?
Maybe we can reduce the size of the data folder when we do some assumptions?

The current diff is pretty much only docs. But there is one super small change in composer.json.

Currently master contains a fix to a bug that makes my CI red. Could I please ask for a new patch release?

Example: Is this correct?

"vásárló" -> vasarlo"

Can a native speaker, please check the character-replacement, thanks.

https://github.com/voku/portable-ascii/blob/master/src/voku/helper/data/ascii_by_languages.php#L274

Example: Is this correct?

"университет" -> "universitet"

Can a native speaker, please check the character-replacement, thanks.

https://github.com/voku/portable-ascii/blob/master/src/voku/helper/data/ascii_by_languages.php#L548

Example: Is this correct?

"sveučilišt" -> "sveucilist"

Can a native speaker, please check the character-replacement, thanks.

https://github.com/voku/portable-ascii/blob/master/src/voku/helper/data/ascii_by_languages.php#L412

Some ASCII character mappings include a quote symbol:

Were these intended?

Example: Is this correct?

"zákazník" -> "zakaznik"

Can a native speaker, please check the character-replacement, thanks.

https://github.com/voku/portable-ascii/blob/master/src/voku/helper/data/ascii_by_languages.php#L1018

Example: Is this correct?

"trường học" -> "truong hoc"

Can a native speaker, please check the character-replacement, thanks.

https://github.com/voku/portable-ascii/blob/master/src/voku/helper/data/ascii_by_languages.php#L723

Example: Is this correct?

"جامعة" -> "gamaa"

Can a native speaker, please check the character-replacement, thanks.

https://github.com/voku/portable-ascii/blob/master/src/voku/helper/data/ascii_by_languages.php#L860

The Turkish character 'i̇' (capital 'I' with dot above) is not currently supported.

CVE-2019-8331 - Medium Severity Vulnerability

Vulnerable Library - bootstrap-4.1.3.min.js

The most popular front-end framework for developing responsive, mobile first projects on the web.

Library home page: https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/4.1.3/js/bootstrap.min.js

Path to vulnerable library: /vendor/phpunit/php-code-coverage/src/Report/Html/Renderer/Template/js/bootstrap.min.js

Dependency Hierarchy:

• ❌ bootstrap-4.1.3.min.js (Vulnerable Library)

Found in HEAD commit: 5f194d2b2d1490498ccef38efc358148119355c9

Found in base branch: master

Vulnerability Details

In Bootstrap before 3.4.1 and 4.3.x before 4.3.1, XSS is possible in the tooltip or popover data-template attribute.

Publish Date: 2019-02-20

URL: CVE-2019-8331

CVSS 3 Score Details (6.1)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Network
  • Attack Complexity: Low
  • Privileges Required: None
  • User Interaction: Required
  • Scope: Changed
• Impact Metrics:
  • Confidentiality Impact: Low
  • Integrity Impact: Low
  • Availability Impact: None

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: twbs/bootstrap#28236

Release Date: 2019-02-20

Fix Resolution: bootstrap - 3.4.1,4.3.1;bootstrap-sass - 3.4.1,4.3.1

Step up your Open Source Security Game with WhiteSource here

CVE-2020-11022 - Medium Severity Vulnerability

Vulnerable Library - jquery-3.3.1.min.js

JavaScript library for DOM operations

Library home page: https://cdnjs.cloudflare.com/ajax/libs/jquery/3.3.1/jquery.min.js

Path to vulnerable library: /vendor/phpunit/php-code-coverage/src/Report/Html/Renderer/Template/js/jquery.min.js

Dependency Hierarchy:

• ❌ jquery-3.3.1.min.js (Vulnerable Library)

Found in HEAD commit: 5f194d2b2d1490498ccef38efc358148119355c9

Found in base branch: master

Vulnerability Details

In jQuery versions greater than or equal to 1.2 and before 3.5.0, passing HTML from untrusted sources - even after sanitizing it - to one of jQuery's DOM manipulation methods (i.e. .html(), .append(), and others) may execute untrusted code. This problem is patched in jQuery 3.5.0.

Publish Date: 2020-04-29

URL: CVE-2020-11022

CVSS 3 Score Details (6.1)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Network
  • Attack Complexity: Low
  • Privileges Required: None
  • User Interaction: Required
  • Scope: Changed
• Impact Metrics:
  • Confidentiality Impact: Low
  • Integrity Impact: Low
  • Availability Impact: None

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://blog.jquery.com/2020/04/10/jquery-3-5-0-released/

Release Date: 2020-04-29

Fix Resolution: jQuery - 3.5.0

Step up your Open Source Security Game with WhiteSource here

Example: Is this correct?

"kundestøtte" -> "kundestoette"

Can a native speaker, please check the character-replacement, thanks.

https://github.com/voku/portable-ascii/blob/master/src/voku/helper/data/ascii_by_languages.php#L714

CVE-2019-11358 - Medium Severity Vulnerability

Vulnerable Library - jquery-3.3.1.min.js

JavaScript library for DOM operations

Library home page: https://cdnjs.cloudflare.com/ajax/libs/jquery/3.3.1/jquery.min.js

Path to vulnerable library: /vendor/phpunit/php-code-coverage/src/Report/Html/Renderer/Template/js/jquery.min.js

Dependency Hierarchy:

• ❌ jquery-3.3.1.min.js (Vulnerable Library)

Found in HEAD commit: 5f194d2b2d1490498ccef38efc358148119355c9

Found in base branch: master

Vulnerability Details

jQuery before 3.4.0, as used in Drupal, Backdrop CMS, and other products, mishandles jQuery.extend(true, {}, ...) because of Object.prototype pollution. If an unsanitized source object contained an enumerable proto property, it could extend the native Object.prototype.

Publish Date: 2019-04-20

URL: CVE-2019-11358

CVSS 3 Score Details (6.1)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Network
  • Attack Complexity: Low
  • Privileges Required: None
  • User Interaction: Required
  • Scope: Changed
• Impact Metrics:
  • Confidentiality Impact: Low
  • Integrity Impact: Low
  • Availability Impact: None

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-11358

Release Date: 2019-04-20

Fix Resolution: 3.4.0

Step up your Open Source Security Game with WhiteSource here

CVE-2020-11023 - Medium Severity Vulnerability

Vulnerable Library - jquery-3.3.1.min.js

JavaScript library for DOM operations

Library home page: https://cdnjs.cloudflare.com/ajax/libs/jquery/3.3.1/jquery.min.js

Path to vulnerable library: /vendor/phpunit/php-code-coverage/src/Report/Html/Renderer/Template/js/jquery.min.js

Dependency Hierarchy:

• ❌ jquery-3.3.1.min.js (Vulnerable Library)

Found in HEAD commit: 5f194d2b2d1490498ccef38efc358148119355c9

Found in base branch: master

Vulnerability Details

In jQuery versions greater than or equal to 1.0.3 and before 3.5.0, passing HTML containing elements from untrusted sources - even after sanitizing it - to one of jQuery's DOM manipulation methods (i.e. .html(), .append(), and others) may execute untrusted code. This problem is patched in jQuery 3.5.0.

Publish Date: 2020-04-29

URL: CVE-2020-11023

CVSS 3 Score Details (6.1)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Network
  • Attack Complexity: Low
  • Privileges Required: None
  • User Interaction: Required
  • Scope: Changed
• Impact Metrics:
  • Confidentiality Impact: Low
  • Integrity Impact: Low
  • Availability Impact: None

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://github.com/jquery/jquery/security/advisories/GHSA-jpcq-cgw6-v4j6,https://github.com/rails/jquery-rails/blob/master/CHANGELOG.md#440

Release Date: 2020-04-29

Fix Resolution: jquery - 3.5.0;jquery-rails - 4.4.0

Step up your Open Source Security Game with WhiteSource here

Example: Is this correct?

"Ελληνικό αλφάβητο" -> "Ellhniko alfabhto"

Can a native speaker, please check the character-replacement, thanks.

https://github.com/voku/portable-ascii/blob/master/src/voku/helper/data/ascii_by_languages.php#L4

Example: Is this correct?

"IGÅR" -> "IGaR"

Can a native speaker, please check the character-replacement, thanks.

https://github.com/voku/portable-ascii/blob/master/src/voku/helper/data/ascii_by_languages.php#L185

Example: Is this correct?

"vaikų darželis" -> "vaiku darzelis"

Can a native speaker, please check the character-replacement, thanks.

https://github.com/voku/portable-ascii/blob/master/src/voku/helper/data/ascii_by_languages.php#L693

Example: Is this correct?

"діти" -> "diti"

Can a native speaker, please check the character-replacement, thanks.

https://github.com/voku/portable-ascii/blob/master/src/voku/helper/data/ascii_by_languages.php#L538

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update actions/cache action to v2.1.8
• chore(deps): update shivammathur/setup-php action to v2.31.1
• chore(deps): update actions/cache action to v4
• chore(deps): update actions/checkout action to v4
• chore(deps): update actions/upload-artifact action to v4
• chore(deps): update codecov/codecov-action action to v4
• chore(deps): update dependency phpunit/phpunit to v11
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Example: Is this correct?

"лысая гора" -> "lysaja gora"

Can a native speaker, please check the character-replacement, thanks.

https://github.com/voku/portable-ascii/blob/master/src/voku/helper/data/ascii_by_languages.php#L468

Example: Is this correct?

"français" -> "francais"

Can a native speaker, please check the character-replacement, thanks.

https://github.com/voku/portable-ascii/blob/master/src/voku/helper/data/ascii_by_languages.php#L1055

Example: Is this correct?

"دانشگاه" -> "danshgah"

Can a native speaker, please check the character-replacement, thanks.

https://github.com/voku/portable-ascii/blob/master/src/voku/helper/data/ascii_by_languages.php#L907

According to your description:

For example, German will map 'ä' to 'ae', while other languages will simply return e.g. 'a'.

this is wrong.
Example:
1 Apfel ( 1 Aple ) > nothing to convert
2 Äpfel (2 Apples) > you will convert Ä to A which is wrong because with an A only the 2 Apples become 1 Apple.
Maybe this sample is a bit hard to understand, but 1 Apfel = 1 Apple , 2 (or more) Äpfel are more Apples.
The only difference is the first letter, either an A or an Ä

Same will be with your sample Düsseldorf, converting it to en for example will translate Düsseldorf to Dusseldorf where it should be in all other languages than German Duesseldorf!
Dusseldorf may also exist as a town somewhere in Europe, if not Dussel means something very special in German (some stupid).

Translating German into other languages, especially for SEO-URLs is not that easy as it looks!
Please recheck and rework your framework, if it will handle such cases also correct, it could be useful.

Example: Is this correct?

"görüşmek" -> "gorusmek"

Can a native speaker, please check the character-replacement, thanks.

https://github.com/voku/portable-ascii/blob/master/src/voku/helper/data/ascii_by_languages.php#L193

Example: Is this correct?

"grădiniță" -> "gradinita"

Can a native speaker, please check the character-replacement, thanks.

https://github.com/voku/portable-ascii/blob/master/src/voku/helper/data/ascii_by_languages.php#L621

I've replaced (locally) Laravel's Str::ascii with a call to this package, however some of the results are not quite as expected. For example, the following test fails:

$this->assertSame('h H sht SHT a A y Y', Str::ascii('х Х щ Щ ъ Ъ ь Ь', 'bg'));

Failed asserting that two strings are identical.

--- Expected
+++ Actual
@@ @@
-'h H sht SHT a A y Y'
+'h H sht Sht a A  '

Input: х Х щ Щ ъ Ъ ия йо

Output: h H sht Sht a A iya yo

Expected output: h H sht Sht a A ia yo

Example: Is this correct?

"တတျနိုငျသ" -> "ttyanongyath"

Can a native speaker, please check the character-replacement, thanks.

https://github.com/voku/portable-ascii/blob/master/src/voku/helper/data/ascii_by_languages.php#L294

Example: Is this correct?

"bērnudārzs" -> "bernudarzs"

Can a native speaker, please check the character-replacement, thanks.

https://github.com/voku/portable-ascii/blob/master/src/voku/helper/data/ascii_by_languages.php#L668

Example: Is this correct?

"универзитет" -> "univerzitet"

Can a native speaker, please check the character-replacement, thanks.

https://github.com/voku/portable-ascii/blob/master/src/voku/helper/data/ascii_by_languages.php#L926

In the process of updating several repos, will link pull request to this issue.

Example: Is this correct?

"пример" -> "primer"

Can a native speaker, please check the character-replacement, thanks.

https://github.com/voku/portable-ascii/blob/master/src/voku/helper/data/ascii_by_languages.php#L208

Hi there!

In case you missed it, I'm working on abandoning patchwork/utf8 in favor of symfony/symfony#33553 In this PR, I'm using transliterator_transliterate() to implement UTF-8 to ASCII transliterations (see AbstractUnicodeString::ascii()). But when the intl extension is not installed, I just skip the call and fallback to NFKC + maps + iconv.

It would be super cool if we could polyfill this function from intl (a subset of its behavior at least). This package is the closest I know. Would you consider porting it to https://github.com/symfony/polyfill/tree/master/src/Intl?

vendor\voku\portable-ascii\src\voku\helper\data\x0fd.php I don't understand the Arabic prayers in some of the files, why did you put them there?

Is this package doing ASCII char handling or is it praying that our project doesn't crash?