Comments (3)
@holopoj ,
Preparing the text for the trie before adding and before searching is a good workaround. I will work with Basic Multilingual Plane which contains characters for almost all modern languages, and a large number of symbols:
/// <summary>
/// It Removes diacritics from text, converts it to lower, removes surrogate
/// characters and normalizes it to prepare text for accent and case insensitive search
/// </summary>
/// <param name="text"></param>
/// <returns></returns>
static string PrepareForTrie(string text)
{
//return text;
var normalizedString = text.Normalize(NormalizationForm.FormD);
var stringBuilder = new StringBuilder();
for (int i = 0; i < normalizedString.Length; i++)
{
char c = normalizedString[i];
var unicodeCategory = CharUnicodeInfo.GetUnicodeCategory(c);
if (char.IsHighSurrogate(c) || char.IsLowSurrogate(c)) continue;
if (unicodeCategory != UnicodeCategory.NonSpacingMark && unicodeCategory != UnicodeCategory.Control)
{
stringBuilder.Append(char.ToLower(c));
}
}
return stringBuilder.ToString().Normalize(NormalizationForm.FormC);
}
Now this code works, it shows 2 and 1. The second item still has the double code point grapheme:
var a = PrepareForTrie("Rosalia de Castro");
var b = PrepareForTrie("rosalía");
var t = new UkkonenTrie<int>(3);
t.Add(a, 1);
t.Add(b, 2);
foreach (var value in t.Retrieve(b))
{
Console.WriteLine(value);
}
from trienet.
The proper fully compatible solution that would resolve most if not all issues with Unicode is to rewrite all substring handling to use the StringInfo
class to work with 'real' characters, i.e. graphemes, rather than individual char
codepoints.
However, the public StringInfo
API is very uncomfortable. E.g. you have to manually pump a non-generic IEnumerator
with MoveNext()
to iterate over a string's graphemes. There's no IEnumerable<>
support and thus also no foreach
support.
[EDIT]
It looks like this wouldn't be too difficult of a change with the Ukkonen trie, if you go about it naively and just replace regular SubString()
calls and Length
accesses with StringInfo
-driven equivalents.
The downsides are that it would probably murder atleast construction performance; and that the Node
class will need to hold an IDictionary<string,Edge>
as a grapheme may not fit in a single char
. That last bit means an increase in space taken as well, but luckily it's still bounded. Unicode graphemes aren't endlessly long, iirc.
Might be better off by one-time converting all strings into a dedicated data structure operating at the grapheme level though. That would certainly keep code more maintainable.
from trienet.
For me the thing throws OutOfBoundsExceptions when I even try to construct something that has any special characters in it. And all my sources are in ISO-8859-1.
So it seems this project is useless in any real world application, unless you're dealing with plain ASCII.
from trienet.
Related Issues (12)
- Case sensitivity HOT 2
- Some issue with Unicode characters, maybe HOT 2
- Not an issue - "fork" or contribution? HOT 2
- Incompatible with UWP because of String.Intern()
- Pattern / wildcard search
- Implement Update and Delete HOT 4
- Only for English? HOT 1
- Net standard 2.0 nuget package is not uploaded HOT 4
- Only one search result instead of two
- Binary targeting .NET Framework 4.5?
- Storing UkkonenTrie in Redis
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from trienet.