Improvement suggestion with multiple domains in one single URL. about xurls HOT 6 CLOSED

This is an interesting feature request. I think it shouldn't be on by default; the default behavior to not show overlapping results is I think the most common need, and, like you say, the fastest.

We can't simply try all possible substrings of a found match. For example, that would mean that foobar.com would also give bar.com, oobar.com, and so on. So we need something a bit smarter.

Perhaps we can limit this feature to at most one nested match within a URL's "path" element (including ?queries etc). That might work, but I wonder how common of a need that is.

Finally, I'm not sure how to expose this in the API or in the command-line tool. Any ideas?

from xurls.

I fully agree this should not be the default.
You are right about the "overlapping" problem to consider.

By "default" now, the requested behaviour is using the space etc as "delimiter".
It may be possible to define a option with an argument defining an additional "new" separator.
-s "/" could mean "use / also as a separator"

$ echo "http://www.fakedomain.com/account/legitdomain.com/folder" | bin/xurls -r -s "/"
http://www.fakedomain.com/account/legitdomain.com/folder
legitdomain.com/folder
$

I fully agree too this requirement is not very common.. Was just an idea as I came across this specific case.. ;)

Thank's

from xurls.

This has nothing to do with separators, though. Your suggestion above would mean we'd support overlapping matches, which we don't now. If anything, adding / as an optional separator would just break all paths.

from xurls.

Yes, suggestion is optional overlapping match.

Now:

$ echo "foo/domainB.com/bar" | bin/xurls -r
domainB.com/bar
$
$ echo "domainA.com/foo/domainB.com/bar" | bin/xurls -r
domainA.com/foo/domainB.com/bar
$

In this example, domainB.com is identified only when there is no "overlap".

Suggestion about / is like "replace all / with space/ and check again if something is found:

"domainA.com /foo/domainB.com/bar"
"domainA.com/foo /domainB.com/bar"
"domainA.com/foo/domainB.com /bar"

then "reassemble" removing the extra space (to have the first full path with dir)

from xurls.

What you're suggesting there would be a recursive search; at best, that would be quadratic complexity, and quite a lot of added code for a niche use case.

I've decided to not implement this for now. It should be fairly easy to implement this outside xurls. If you find that the API is nice and the code is generally useful, we can look at adding that later, but for now I don't think it's worth the effort.

from xurls.

That's OK. Thank's

from xurls.