Memory leaks when using IncludeOptimized and BulkSaveChanges together with PooledDbContextFactory [7.0.100] about entityframework-extensions HOT 5 OPEN

Hello @schuettecarsten ,

The IncludeOptimized feature uses the QueryFuture to execute his queries.

We can see that a ConditionalWeakTable is used on this line: https://github.com/zzzprojects/EntityFramework-Plus/blob/master/src/shared/Z.EF.Plus.QueryFuture.Shared/QueryFutureManager.cs#L61C35-L61C139

So, if the context is never disposed of, the QueryFutureBatch related will never be disposed of either and will just continue to grow. That could probably explain the current behavior with the PooledDbContextFactory. However the Queries are cleared
once executed, so it doesn't totally explain why it grows by 60 to 80 MB every run.

Is it possible for you to try in a development environment and see if the same behavior happens with SaveChanges? If that's the case, that will give us a big hint that the troublemaker is probably the IncludeOptimized method.

Best Regards,

Jon

from entityframework-extensions.

Hi @JonathanMagnan,

thank you for the explanation. I have done some tests and the memory leaks go away when I switch from PooledDbContextFactory to DbContextFactory. Using SaveChanges instead of BulkSaveChanges has no effect. I cannot use the standard Include because this will break the software due to excessive memory consumption.

Do not trust the 60 to 80 megabytes value, it's just what I could estimate from task manager while observing the software. I took a memory dump and analyzed it using Jetbrains dotMemory and could see lots of entities held by the DbContext instances that were stored in the pool.

from entityframework-extensions.

Hello @schuettecarsten ,

Thank you for the info, so there is a huge chance that the memory leak is due to the ConditionalWeakTable as the DbContext is never really released.

We will continue to look at it, but you might have a few options you might want to try.

Clearing the cache

Using QueryFutureManager.CacheWeakFutureBatch.Clear(); to clear the ConditionalWeakTable (if you can) from time to time. To free some memory if we indeed have a memory leak on that feature.

https://github.com/zzzprojects/EntityFramework-Plus/blob/master/src/shared/Z.EF.Plus.QueryFuture.Shared/QueryFutureManager.cs#L61

Using IncludeGraph with the most recent version

It will not fix this issue and will require you to use method such as BulkMerge, but I find it worth mentioning as our new IncludeGraph dramatically decreases the memory usage: https://entityframework-extensions.net/v7-100-0-0-include-graph#memory-performance-improvements

If you test the Clear on the ConditionalWeakTable, let us know if that fixed the memory leak

from entityframework-extensions.

@JonathanMagnan
During my tests I found an issue with BulkSaveChangesAsync, when trying to set IncludeGraph to true, I get the exception Oops! You cannot use the IncludeGraph option with LegacyIncludeGraph. Use one or the other but not both, but I did not touch LegacyIncludeGraph at all, only IncludeGraph to true. Setting LegacyIncludeGraph to false did not help.

from entityframework-extensions.

@JonathanMagnan
Calling QueryFutureManager.CacheWeakFutureBatch.Clear() from time to time does not fix the memory issue, unfortunately.

from entityframework-extensions.