foundatiofx / foundatio.repositories Goto Github PK

Some items may fail due to version change or multiple bulk results running at the same time. We should retry this and possibly return the number of modified documents (bulkResult.Items). Also, we should invalidate the cache before returning..

https://github.com/exceptionless/Foundatio.Repositories/blob/master/src/Elasticsearch/Repositories/ElasticRepositoryBase.cs#L178

We need to ensure though that this doesn't affect cache invalidation.

Patternize multi-tenant custom field handling

• Custom field mapping storage
  • Entity type (repo), tenant (should allow for composite like organization+type), field type, friendly name, extensible to store additional mapping information
  • Shared across multiple entity types and tenants
  • Auto assign index slots to new custom fields
  • Fill in gaps for index slots when fields have been deleted
  • Ability to efficiently lookup field mapping for an entity type and tenant
    • Aggressively cache mapping for the tenant
  • Ability to have a list of aliases for a field that will be used to resolve queries
    • Only the root name will be used for getting data that gets added to the idx slot
    • Aliases count as used names can't be used by other fields
• Extensible field types
  • Packaged up field type behaviors that can be re-used
  • Built in field types for string, date, bool, number, float, etc
  • Consumers can create custom field types to add additional indexing formats like address, geo, etc
  • Ability to transform values when adding them to idx
    • Allows the data to be optimized for searching
    • Allows things to happen like looking up related data to store denormalized
    • Probably needs to allow batching of values for efficiency when doing idx transforms
  • Standard field types automatically configured, but can be overridden with custom behavior
  • Ignore errors for invalid document field values when saving
  • Ability to do advanced things like scripted fields that have access to the rest of the document to calculate values (proven out in tests) https://github.com/FoundatioFx/Foundatio.Repositories/blob/master/tests/Foundatio.Repositories.Elasticsearch.Tests/Repositories/Configuration/Indexes/CalculatedIntegerFieldType.cs
• Easily implemented in repositories
  • Interface for model to implement that opts the repository into supporting custom fields
  • Automatically copies values from friendly names to idx slots
  • Automatically gets mapping and sets QueryFieldResolver on queries
• Field management
  • Ability to change type of an existing field
    • Could start out as an auto field indexed as string and then be changed to a date or some other more specific type
  • Ability to remove a field
    • Cleanup idx values
    • Optionally remove custom field values
  • Ability to rename a field
• Ability to auto create custom fields on the fly when we see a new value that does not have a mapping
  • Default to store as string with keyword subfield
  • Hook when auto creating custom property mappings that lets you decide if a mapping should be created and determine the default index type

I removed some usages of JSON.NET down to where our only usage is pretty much the JSON patch support.

Spent some time taking a look at json-everything's JsonPatch support and it only uses JSON Pointers for paths and does not support some of the things we are doing where we are allowing full JSON Path support where we can do things like $.books[?(@.author == 'John Steinbeck')] to remove all array elements from the books array that has an author sub-property that matches John Steinbeck. So basically we would need to make major changes to their libraries or talk them into supporting JSON Path expressions even though the unofficial JSON Patch spec only supports JSON Pointer.

Originally posted by @ejsmith in #96 (comment)

Found this issue where someone asked him and he said no to supporting JSON Path.

Originally posted by @ejsmith in #96 (comment)

Comment for previous history:

For Json Patching it would be nice to see if we can get it to work without a dependency or we can internalize the dependency. I see several spots doing patching

• https://github.com/gregsdennis/json-everything
• https://github.com/ladeak/JsonMergePatch
• .NET 7 is adding patching to System.Net.Http.Json, we could see how they did it and perhaps do it here.

Originally posted by @niemyjski in #96 (comment)

We need to be able to resume reindexing by keeping the last scroll id + a list of document id's in a cache set.

A final request to elastic that is created after parsing a filter expression has a range filter but without a timezone specified.
More info here:
https://shusson.info/post/elasticsearch-range-query-datemath-and-timezones

Failing test:

`[Fact]
 public async Task GetWithDateRangeFilterExpressionHonoringTimeZoneAsync() {
            Log.MinimumLevel = LogLevel.Trace;
            var chicagoTimeZone = TZConvert.GetTimeZoneInfo("America/Chicago");
            var asiaTimeZone = TZConvert.GetTimeZoneInfo("Asia/Shanghai");
            var format = @"yyyy-MM-ddTHH\\:mm\\:ss";
            var utcNow = SystemClock.UtcNow;
            var chicagoNow = TimeZoneInfo.ConvertTimeFromUtc(utcNow, chicagoTimeZone);
            var asiaNow = TimeZoneInfo.ConvertTimeFromUtc(utcNow, asiaTimeZone);
            _logger.LogInformation($"UTC: {utcNow:o} Chicago: {chicagoNow:o} Asia: {asiaNow:o}");

            var employee = await _employeeRepository.AddAsync(EmployeeGenerator.Generate(nextReview: utcNow), o => o.ImmediateConsistency());
            Assert.NotNull(employee?.Id);
            var filter = $"next:>={utcNow.SubtractHours(1).ToString(format)} AND next:<={utcNow.ToString(format)}";
            var results = await _employeeRepository.FindAsync(o => o.FilterExpression(filter));
            _logger.LogInformation($"Count: {results.Total} - UTC range");
            Assert.NotNull(results);
            Assert.Equal(1, results.Documents.Count);
            Assert.Equal(1, results.Page);
            Assert.False(results.HasMore);
            Assert.Equal(1, results.Total);

            filter = $"next:>={chicagoNow.SubtractHours(1).ToString(format)} AND next:<={chicagoNow.ToString(format)}";
            results = await _employeeRepository.FindAsync(o => o.FilterExpression(filter).BuildOption("timezone", "America/Chicago"));
            _logger.LogInformation($"Count: {results.Total} - Chicago range");
            Assert.NotNull(results);
            Assert.Equal(1, results.Documents.Count);
            Assert.Equal(1, results.Page);
            Assert.False(results.HasMore);
            Assert.Equal(1, results.Total);

            filter = $"next:>={asiaNow.SubtractHours(1).ToString(format)} AND next:<={asiaNow.ToString(format)}";
            results = await _employeeRepository.FindAsync(o => o.FilterExpression(filter).BuildOption("timezone", "Asia/Shanghai"));
            _logger.LogInformation($"Count: {results.Total} - Asia range");
            Assert.NotNull(results);
            Assert.Equal(1, results.Documents.Count);
            Assert.Equal(1, results.Page);
            Assert.False(results.HasMore);
            Assert.Equal(1, results.Total);
        }`

Foundatio.Repositories Support entity farmework??

For example, we need to refresh the indexes after we do soft deletes on events so the stacks can be deleted properly.

There could be added documents, so should we also send out a Added ChangeType notification? I almost think that we should have an event handler for Documents Saved and DocumentsAdded that just handles notifications. This would ensure that it's always executed properly.

https://github.com/exceptionless/Foundatio.Repositories/blob/master/src/Elasticsearch/Repositories/ElasticRepositoryBase.cs#L280-L281

Currently aggregations can only return number results. We need to support complex aggregations (geo + string data etc).

Currently users of this library using System.Text.Json (STJ) cannot return search aggregation results (from redis cache, or from api endpoints) due to serialization dependencies on Newtonsoft.Json. Our goal is to update the Foundatio.Repositories to work with System.Text.Json. However, we don’t want to break other applications so we need to support Newtonsoft.Json by leaving the existing JsonConverters in place unless we can find a way to support both out of the box without the dependency or an easy way for it to be configured externally.

I see this happening in the following steps:

1. Copy the following code snippet to end of AggregationQueryTests.GetTermAggregationsWithTopHitsAsync().

            string systemTextJson = System.Text.Json.JsonSerializer.Serialize(result);
            Assert.Equal(json, systemTextJson);
            roundTripped = System.Text.Json.JsonSerializer.Deserialize<CountResult>(systemTextJson);
            Assert.Equal(10, roundTripped.Total);
            Assert.Equal(1, roundTripped.Aggregations.Count);
            Assert.Equal(10, roundTripped.Aggregations.Terms<int>("terms_age").Buckets.Count);
            bucket = roundTripped.Aggregations.Terms<int>("terms_age").Buckets.First(f => f.Key == 19);
            Assert.Equal(1, bucket.Total);

2. Add custom converter to handle writing polymorphic json. The nest team is doing something like this: https://github.com/elastic/elasticsearch-net/blob/main/src/Elastic.Clients.Elasticsearch/_Generated/Types/Aggregations/TermsAggregation.g.cs#L27 for one of there aggretaions. I think you'd pass this converter in options of the first call to Serialize in the snippet above. This will solve the first assert.
3. You'd need to add and convert the existing JsonConverters in the Foundatio.Repositories.Utility to System.Text.Json and then add the json converter attributes next to the existing converter attributes. This will fix the round trip part of reading the json text.
4. Request PR Review

Not sure if this link will help on the converters: https://bengribaudo.com/blog/2022/02/22/6569/recursive-polymorphic-deserialization-with-system-text-json