foundatiofx / foundatio Goto Github PK
View Code? Open in Web Editor NEWPluggable foundation blocks for building distributed apps.
License: Apache License 2.0
Pluggable foundation blocks for building distributed apps.
License: Apache License 2.0
You can provide the IJob interface and some different job implementations that derive from
As mentioned by @moswald in #38 ...
EnqueueAsync()
and Thread B calls DeleteQueueAsync()
EnsureQueueCreatedAsync()
, ensuring _queueClient
is not null_queueClient
to null_queueCleint
which is now nullThere's not really a nice way to handle this without locking. Couple of solutions:
Wrap _queueClient
in a property which throws a slightly different exception, leaving the user to retry.
Implement retry logic (preferably in QueueBase
).
Add reentrant read-write lock to get better performance while doing read only operations.
https://github.com/Netflix/curator/wiki/Shared-Reentrant-Read-Write-Lock
It's like this ?
somecode to get lockprovider
var rediscco = new RedisCacheClientOptions() { ConnectionMultiplexer = muxer };
_lockProvider = new CacheLockProvider(new RedisCacheClient(rediscco), new RedisMessageBus(new RedisMessageBusOptions() { Subscriber = rediscco.ConnectionMultiplexer.GetSubscriber() }));
some code to get lock
public static async Task<ILock> AcquireAsync(string name,
TimeSpan? lockTimeout = null,
CancellationToken cancellationToken = default(CancellationToken),
bool continueOnCapturedContext = false)
{
var @lock = await _lockProvider.AcquireAsync(name, lockTimeout, cancellationToken)
.ConfigureAwait(continueOnCapturedContext);
return @lock;
}
user lock
var l = LockHelper.AcquireAsync(key, TimeSpan.FromSeconds(5)).Result ;
............my code to process stock............
l.ReleaseAsync();
I'm looking to be able to use some of the great functionality that I just found in the Foundatio library.
However, we have an application that we currently support on .net 4.5.2. It seems that a few months ago there was a change that Foundatio only supports 4.6 and later.
Is my only choice to use v4.1? Any plans to backport fixes to this branch for people who can't move ahead to 4.6 at this time?
Thanks.
Hi
We have a scenario where we need to support duplicate detection for messages being enqueued which is done by messageid https://msdn.microsoft.com/library/azure/microsoft.servicebus.messaging.brokeredmessage.messageid.aspx
I've done the basics in a fork (dpvreony-forks@50f18ce) but it is currently off the back of #71
Thanks
We don't send a progress message when a work item fails. What should the status be?
Love the building blocks!
It is not a bug, it is a feature request
It would be really cool, if there could be an implementation of ICacheClient via System.Web.Caching.Cache
When AzureServiceBusMessageBus
receives a poisonous message, it completes it with a log entry. ASB supports deadletter queues. Poisonous messages should be forwarded to the DLQ rather being completed.
We currently retrieve all records for FileStorage and don't implement paging at all for AzureFileStorage. We would need to figure out how to do custom paging with ListBlobsSegmentedAsync
Hi
We have a need to support https://www.nuget.org/packages/ServiceBus.v1_1/ I have done a fork (master...dpvreony-forks:master), it's pretty much a carbon copy of the existing Azure project. I can raise a PR if you're happy.
Thanks
We need to find a good way to log the work item data when a work item fails
Already implemented in Azure Service Bus and Storage Queues.
Redis: https://github.com/exceptionless/Foundatio/src/Redis/Queues/RedisQueue.cs#L297
InMemory: https://github.com/exceptionless/Foundatio/blob/master/src/Core/Queues/InMemoryQueue.cs#L174
Obviously there are many more "Azure ServiceBus Queue" specific properties that are unsupported, but these 2 are the ones I need right now. Do you guys have a strategy for how you might support these types of properties moving forward?
I started to do a pull request to set the properties via the constructor, but bolting on properties piecemeal seems like an unsustainable strategy.
Right now, EnsureQueueCreatedAsync
will check if the queue exists, and then if not, it will try to create it. If another process creates the queue in-between these two calls, then the create fails with a MessagingEntityAlreadyExistsException
. This is caught and handled and works fine.
However, there's a catch - if another process is currently creating the queue, then it will fail with MessagingException
, which is not caught.
The error seen in logs is:
Microsoft.ServiceBus.Messaging.MessagingException: The remote server returned an error: (409) Conflict. SubCode=40901. Another conflicting operation is in progress.
Since MessagingEntityAlreadyExistsException
derives from MessagingException
, the solution is to catch MessagingException
instead.
Would it be possible for you to publish strong named nuget packages which foundation depdending on strong named/signed package when they are available, for instance StackExchange.Redis.StrongName. This would be helpful for using Foundation along other packages such as SignalR.Redis.
I hope you will consider doing this.
The FolderFileStorage class calls stream.Seek(0, SeekOrigin.Begin) on the passed stream before copying the contents to the file stream. While this might be convenient (in some cases), it is inconsistent with the other implementations of IFileStream (InMemoryStream and AzureFileStream). And it seems presumptive to assume the caller did not positioned the stream correctly for the save operation and to "fix it for them". I can't think of any stream handling functions that do this - including the underlying ones called by these classes: Stream.CopyTo() and CloudBlockBlob.UploadFromStream().
My recommendation is to remove the Seek() operation.
We've recently added the ability to renew the lock on the queue entry.
It would be great if we could (optionally) do this automatically in QueueProcessorJobBase for long running processes. The alternative would be to pass the queue entry around everywhere to renew - including in scenarios where you don't care if the code has been triggered from a queue or otherwise.
We should be using the new project.json file format and target cross plat.
Similar to the logging and metrics (probably), it would be useful to have an eventqueue that all of your classes/methods publish to (in the situations where someone initializes it). Then subscribers of the event queue could listen for certain events and perform actions based on it.
This concept could basically leverage existing components you have already. And I haven't looked at your all of your code yet, but maybe this is how your metrics already works.
The use case for this would be something like a Job that subscribes for all SaveFile events... and as soon as it happens, it runs a command. But that same Job may run for DeleteFile events. And so, it can just subscribe to that event too. I know this feature is something that easily could be built into a custom project utilizing other foundatio components. However, I think it would be great to be a foundational pipeline used by your framework.
Currently we are abandoning queue items when there is an error which is great. But in some scenarios this doesn't work the greatest for example. When the queue item has been abandoned more times then the retry limit, there is no way to see when the queue item will be removed. As such we can't clean up data data payloads on disk that the queue item points to.
The library Metrics.Net is only available for netfx and only can be deployed on windows servers. But the Foundatio currently supports netstandard 2.0, the Metrics.Net should be replaced with AppMetrics which supports netstandard, so that we can deployed on linux or unix.
A lot of the Microsoft.ServiceBus.NamespaceManager
methods called in the AzureServiceBusQueue
constructor have async
counterparts. Since constructors can't be async
, I propose deprecating the current constructor (shakes fist at semver) and adding a new private one that does very little. Then introduce a new factory method: static async Task<AzureServiceBusQueue>
factory method that does all the same work the current constructor does synchronously.
This would allow consumers to know when it's reached it's max retry account and will be discarded.
We could possibly do this by adding a new event for when an item is placed into DeadLetter.
Hey,
Great foundations here. I'd be tempted to use this for a project i'm working on (which currently just uses the standard ServiceBus nuget) however we're using the Topics and Subscription functionality.
The AzureServiceBusMessageBus
implementation uses the subscription to determine the channel to push to the multiple subscribers however there doesn't seem to be a way to listen to individual subscriptions being exposed. Would there be any interest in this functionality being added?
For the roadmap you could mirror the go-kit roadmap.
IMessageBus messageBus = new RedisMessageBus(new RedisMessageBusOptions
{
Subscriber = connectionMultiplexer.GetSubscriber()
});
await messageBus.SubscribeAsync<SimpleMessage>(msg =>
{
Console.WriteLine(msg.Data);
});
await messageBus.PublishAsync(new SimpleMessage { Data = "Hello" });
It looks like in the Job sample in the documentation
https://github.com/FoundatioFx/Foundatio#sample-4
The parameter to RunInternalAsync should be of type JobContext to match
https://github.com/FoundatioFx/Foundatio/blob/master/src/Foundatio/Jobs/JobBase.cs
The current implementations of messaging use message pumps with callbacks which is not as efficient as bulk requesting in some circumstances. High volume scenarios should be bulk fetching to reduce costs as well since you are charged per request.
Publish and subscribe should ideally have corresponding bulk overload/versions as well.
in my case i both use java and .net ,in the messagebusbase class ,
field: _knownMessageTypesCache
method: GetMessageBodyType,SendMessageToSubscribers
in java the type is different from .net
so i want create an attribute in both java and .net ,use the _knownMessageTypesCache to mapping the type to class in java and to type in .net .
please change field protected and methods virtual,thanks
the code is in https://github.com/FoundatioFx/Foundatio/blob/master/src/Foundatio/Messaging/MessageBusBase.cs
Throwing any exception in a tight loop/continuously running code path is not a good idea and can really cause performance issues. I think we should return null instead.
Some items may fail due to version change or multiple bulk results running at the same time. We should retry this and possibly return the number of modified documents (bulkResult.Items). Also, we should invalidate the cache before returning..
Would it be possible to create strong name versions of the nuget packages? I am using Signal.Redis which has a dependency on StackExchange.Redis.StrongName while Foundation.Redis has a dependency on StackExchange.Redis. I can build the Foundation solution with the proper dependency but I would imagine I am not the only one having this issue and the community could benefit from such strong named nuget packages. Thank you.
When the value of parameter 'compress' is set true, it should serialize/deserialize with MessagePack.LZ4MessagePackSerializer.NonGeneric
, otherwise should use MessagePack.MessagePackSerializer.NonGeneric
I use log4net in my project, so I wonder if Foundatio.Loggging will have a log4net appder?
There could be added documents, so should we also send out a Added ChangeType notification? I almost think that we should have an event handler for Documents Saved and DocumentsAdded that just handles notifications. This would ensure that it's always executed properly.
Our current implementation of IQueue
has a DeleteQueueAsync method which actually recreates the queue.
This clearly isn't expected behaviour (meaning we can't delete the queue easily in the test tear town) and in addition, we are also performing remote calls in the constructor of the queue which isn't ideal.
We could introduce an additional interface to manage the queues, although we'd have to specifically create every queue and inject it into the container - and the bootstrapper class therefore must ideally be async compatible.
public interface IQueueManager<TQueue> where TQueue : IQueue<T> {
Task<TQueue> CreateQueueAsync(string queueName /* other params */);
Task<bool> QueueExistsAsync(string queueName);
Task DeleteQueueAsync(string queueName);
}
A big down side to this would be it would also make it more difficult to specify additional constructor params for the queue (as used in AzureStorageQueue to control poll interval).
https://docs.microsoft.com/en-us/aspnet/core/fundamentals/logging
Microsoft is using their simple standardized and expandable logging library - https://github.com/aspnet/Logging.
It would be great if you implement some sort of integration between Foundatio's logging library and Microsoft's one to see foundatio log messages in unified way.
Adding a cache tagging feature would allow us to invalidate keys when we otherwise wouldn't be able to.
Recently when deploying exceptionless, in order to improve the scale out ability, we want to make use of cloud storage. Currently the Foundatio only support windows azure, but the Aliyun is with higher efficiency in China.
When we want to request pr to implement the adapter between Foundation and Aliyun OSS, there is no project for this purpose, and we don't know how to request pr for it.
I'm trying to cache an ecommerce cart items using InMemoryCacheClient (dev) and in the future RedisCacheClient.
My key is in the form S{sessionId}-V{productId} and the value contains a CartItem object (quantity and other related data)
Adding and removing items are working great but i need to retrieve all the items for a sessionId.
I know the existance of using the Set but i needed the lines separated.
Is there a way to perform this? Maybe by using the ScopedCacheClient (i've done it with InMemoryCacheClient through the Keys prop but it will not work on RedisCacheClient)
This would allow you to scope all the metric names easily.
Hi all!,
I'm trying to install package on project with .NET 4.5.2. and when
Install-Package Foundatio -Version 4.2.1183
I've this error
Executing nuget actions took 1,94 sec
Install-Package : Could not install package 'Foundatio 4.2.1183'. You are trying to install this package into a project that targets '.NETFramework,Version=v4.5.2', but the package does not contain any assembly
references or content files that are compatible with that framework. For more information, contact the package author.
At line:1 char:1
+ Install-Package Foundatio -Version 4.2.1183
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Install-Package], Exception
+ FullyQualifiedErrorId : NuGetCmdletUnhandledException,NuGet.PackageManagement.PowerShellCmdlets.InstallPackageCommand
Why?
Just like there's storage support for Azure Blob and S3, it would be quite nice to have the opportunity to be cloud-agnostic in what regards to Queues.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.