URLs containing unicode are not matched correctly about xurls HOT 5 CLOSED

This is definitely not an issue regarding non-ascii chars, as they are supported: http://**.**/** matches correctly.

I'm limiting the amount of symbols allowed in urls though, to keep symbols often used in shells and programming languages to be eaten. I'll see if I can add a unicode category to get this to work.

from xurls.

This might be related but I couldn't get it to match https://shmibbles.me/tmp/自殺でも？.png. xurls gives me https://shmibbles.me/tmp/自殺でも when run directly. In conjunction with st's external pipe it doesn't go past the first unicode character. But your test case still works.

from xurls.

@j605 do you happen to know what unicode category ？ is in?

from xurls.

http://graphemica.com/%EF%BC%9F (“Other Punctuation” apparently)

from xurls.

@HalosGhost you're right, thank you.

Here is the full list: http://www.fileformat.info/info/unicode/category/Po/list.htm

We already allow many of the ascii ones in there like ., ,, ! and ? itself as long as they are not at the end of the URL. It makes things more consistent to do this across the entire unicode character class.

I'm applying this change. The only breaking change that my tests catch is that " is now allowed if not in the end of the URL. So before, http://foo.com/@"style="color:red"onmouseover=func() would match just http://foo.com/ but now it matches all of it (I think I found this one in some twitter tests).

This example is very obscure though, and I'd rather be consistent and support unicode better than support these very rare edge cases.

from xurls.