Comments (3)
I added another cache on top of the parallel.
Seems working fine 90% of the time.
It would fail intermittently.
public List<DmnDecisionResult> ExecuteDmnUsingArrayParamParallel(string dmnXmlString, string decisionName, string jsonArrayParam)
{
ConcurrentDictionary<string, DmnDecisionResult> cache = new();
ConcurrentBag<DmnDecisionResult> results = new();
var model = DmnParser.ParseString13(dmnXmlString);
var definition = DmnDefinitionFactory.CreateDmnDefinition(model);
_ = DmnExecutionContextFactory.CreateExecutionContext(definition).ExecuteDecision(decisionName);
JsonDocument.Parse(jsonArrayParam).RootElement
.EnumerateArray()
.AsParallel()
.ForAll(param =>
{
var hash = param.GetRawText();
if (cache.ContainsKey(hash))
{
results.Add(cache[hash]);
return;
}
var ctx = DmnExecutionContextFactory.CreateExecutionContext(definition);
foreach (var property in JsonDocument.Parse(param.GetRawText()).RootElement.EnumerateObject())
{
dynamic propertyvalue = getPropertyValue(property.Value);
ctx.WithInputParameter(property.Name.Trim().Replace(' ', '_'), propertyvalue);
}
var result = ctx.ExecuteDecision(decisionName);
cache.TryAdd(hash, result);
results.Add(result);
});
return results.ToList();
}
from common.dmn.engine.
Hello @NanFengCheong ,
well, the DMN Engine was not designed for parallel processing, although it's interesting idea.
The issue is in DmnExecutionContext class. It uses static Dictionary
as cache for pre processed expressions:
private static readonly Dictionary<(string, Type), Lambda> ParsedExpressionsCache = new Dictionary<(string, Type), Lambda>();
That's, I guess, where you're getting the concurrent access exception (while updating the cache during processing).
I did a quick test just by changing the Dictionary
to ConcurrentDictionary
and it seems to be working (and it also passes all tests):
private static readonly ConcurrentDictionary<(string, Type), Lambda> ParsedExpressionsCache = new ConcurrentDictionary<(string, Type), Lambda>();
So, you can try this "hack", however, as mentioned above, it was not designed this way, so there might be some other consequences (none came to my mind now, but...).
However, there is a better way how to solve (or workaround) your issue I think:
I built a bigger dataset in test code you provided to get a more demanding load:
string jsonString = "{\"TypeOfDay\":\"Holiday\",\"WeatherInCelsius\":24},{\"TypeOfDay\":\"Weekend\",\"WeatherInCelsius\":4}";
for (var i = 0; i < 15; i++)
{
jsonString = $"{jsonString},{jsonString}";
}
jsonString = $"[{jsonString}]";
XML parsing and creation of model definition (part of the DmnExecutionContextFactory.CreateExecutionContext
overload you use) are quite compute heavy operations, that can be run only once and just the "pure" context is needed for each parallel item. So it will be better to do it this way:
var model = DmnParser.ParseString13(DmnXmlString);
var definition = DmnDefinitionFactory.CreateDmnDefinition(model);
System.Text.Json.JsonDocument.Parse(jsonString).RootElement
.EnumerateArray()
.AsParallel()
.ForAll(param =>
{
var ctx = DmnExecutionContextFactory.CreateExecutionContext(definition);
foreach (var property in System.Text.Json.JsonDocument.Parse(param.GetRawText()).RootElement.EnumerateObject())
{
dynamic propertyvalue = getPropertyValue(property.Value);
ctx.WithInputParameter(property.Name.Trim().Replace(' ', '_'), propertyvalue);
}
var result = ctx.ExecuteDecision("Dish Decision");
results.Add(result);
});
Even with your simple model and execution of 65536 decisions in parallel (the "bigger dataset" mentioned above), the processing time has been decreased from about 24s to 14s at my machine.
This will also bring the possibility to get rid of the issue with the parallel access to the expressions cache (so no need to modify the DmnExecutionContext class). Just build the cache before running the parallel block, so there will be no further (parallel) updates.
I believe, if your code will look like this,
var model = DmnParser.ParseString13(DmnXmlString);
var definition = DmnDefinitionFactory.CreateDmnDefinition(model);
var ctxTmpBuildCache = DmnExecutionContextFactory.CreateExecutionContext(definition);
var resultDummy = ctxTmpBuildCache.ExecuteDecision("Dish Decision");
System.Text.Json.JsonDocument.Parse(jsonString).RootElement
.EnumerateArray()
.AsParallel()
.ForAll(param =>
{
var ctx = DmnExecutionContextFactory.CreateExecutionContext(definition);
foreach (var property in System.Text.Json.JsonDocument.Parse(param.GetRawText()).RootElement.EnumerateObject())
{
dynamic propertyvalue = getPropertyValue(property.Value);
ctx.WithInputParameter(property.Name.Trim().Replace(' ', '_'), propertyvalue);
}
var result = ctx.ExecuteDecision("Dish Decision");
results.Add(result);
});
it will not only run faster, but it will also solve the original issue with concurrent update of cache
Hope this help
Radek
from common.dmn.engine.
Hi @adamecr ,
Appreciate for the quick reply.
I managed to get it working.
Thanks.
from common.dmn.engine.
Related Issues (15)
- NormalizeVariableName(string name) HOT 7
- DMN "CodeCoverage" HOT 2
- XML file HOT 2
- Memory leak detected within DmnExecutionContext class HOT 14
- How run all decisions ? HOT 2
- Issue in net 6 with parsing definition from DM HOT 7
- duration nog conforming to FEEL specification HOT 1
- Support for DMN 1.3 HOT 13
- FEEL instead of SFEEL HOT 3
- how do i install DMN.Engine in my .NET project? HOT 5
- DMN files from latest version of camunda modeler not working HOT 4
- DmnExecutionContext.ExecuteDecision() seems to be using the label instead of the name to identify output from a dependent Decision HOT 2
- ParsedExpressionsCache Corruption when running multiple concurrent evaluations HOT 5
- Return matched rules as part of Decision Result HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from common.dmn.engine.